Panoramic image processing method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure relate to a panoramic image processing method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring target images respectively collected by a plurality of cameras, the plurality of cameras having different distortion parameters; determining, from a pre-calibrated association relationship between the cameras and the distortion parameters, a distortion parameter corresponding to each camera; on the basis of the distortion parameters respectively corresponding to the cameras, performing distortion correction on the target images respectively collected by said cameras, and obtaining a panoramic image by fusing all the corrected images. In the panoramic image processing method of the embodiments of the present disclosure, distortion correction and stitching processing can be performed on a 360-degree surround-view module having mixed different cameras, thereby achieving good compatibility.

Description

Panoramic image processing methods, devices, electronic equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202510043725.7, filed on January 10, 2025, entitled "Panoramic Image Processing Method, Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to, but is not limited to, the field of image processing technology, such as a panoramic image processing method, apparatus, electronic device, and storage medium. Background Technology

[0003] Panoramic imaging refers to capturing a wide field of view, typically 360 degrees, using camera equipment. This allows viewers to see the entire surrounding environment in a single image or video. Panoramic imaging is usually based on a camera system using four cameras simultaneously capturing images from different directions, which are then stitched together to create a complete panoramic view. It has applications in automotive, virtual reality (VR), security monitoring, and video conferencing. Its widespread use in the automotive industry is particularly noteworthy, as panoramic imaging (360-degree surround-view cameras) provides a more comprehensive field of view, helping drivers better understand their surroundings, avoid collisions, and enhance driving safety.

[0004] Currently, the main components of a 360° surround view camera device include a lens, a sensor (image sensor), an ISP (Image Signal Processor), a power chip, a storage chip, and SERDES (video transmission chip). Summary of the Invention

[0005] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0006] This disclosure provides a panoramic image processing method to solve the problem of poor compatibility of image processing methods, enabling panoramic camera devices to use cameras from different suppliers for mixed installation; this disclosure also provides a panoramic image processing device; this disclosure also provides an electronic device.

[0007] The technical solution adopted in the embodiments of this disclosure is as follows:

[0008] A panoramic image processing method, the method comprising:

[0009] Acquire target images captured by multiple cameras, each with different distortion parameters;

[0010] The distortion parameters corresponding to each camera are determined from the pre-calibrated correlation between cameras and distortion parameters.

[0011] The distortion of the target image acquired by the camera is corrected according to the distortion parameters of the camera, and all the corrected images are fused together to obtain a panoramic image.

[0012] Based on the aforementioned technical means, product data corresponding to cameras from various suppliers are pre-stored in the storage unit. When calibrating the installed cameras, the product data corresponding to different cameras can be retrieved for calibration. After camera calibration, during power-on use, when processing images captured by different cameras, the distortion parameters, camera intrinsic parameters, and camera extrinsic parameters corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and pre-written product data. The images captured by each camera are then converted, distortion-corrected, and stitched together to obtain a panoramic image. In this way, cameras from different suppliers can be mixed and assembled to form a 360-degree surround-view module.

[0013] The acquisition of target images obtained from multiple cameras includes:

[0014] The product data of the camera is retrieved from the preset storage unit, and the transformation matrix of the original image captured by the camera is determined from the camera coordinate system to the world coordinate system based on the product data of the camera.

[0015] In the world coordinate system, a bowl-shaped model is established with the target point as the origin;

[0016] Based on the transformation matrix, the original image is mapped onto the bowl-shaped model in the world coordinates to obtain the target image. The original images from multiple cameras are all mapped onto the bowl-shaped model to obtain multiple target images.

[0017] Based on the above technical means, the original images are all converted to the same bowl-shaped model. On the one hand, images from different suppliers can be converted to the same coordinate system. On the other hand, the target image is displayed in a bowl-shaped model, making the target image closer to the image captured by the surround-view camera (or fisheye camera). The image distortion correction is more accurate, and the display effect of the stitched image is better.

[0018] The acquisition of target images obtained from multiple cameras includes:

[0019] When a camera is detected to be powered on, the pre-calibrated cameras are queried. The pre-calibrated cameras are associated with distortion parameters and stored in the storage unit.

[0020] When the pre-calibrated cameras include powered-on cameras, the target image captured by the powered-on cameras is obtained. When multiple cameras are powered on, the target images captured by each of the multiple cameras are obtained respectively.

[0021] Based on the above technical means, each time the camera is powered on, it is checked whether the powered-on camera has been calibrated. In the mixed installation scheme of multiple types of cameras, it can ensure that the subsequent image acquisition and image processing processes can proceed normally.

[0022] The step of acquiring target images captured by multiple cameras also includes:

[0023] When it is determined that the pre-calibrated cameras do not include powered-on cameras, the product data corresponding to the powered-on cameras is retrieved from the preset storage unit. The product data includes at least one of the camera's basic information, lens information, camera internal parameters, and camera external parameters.

[0024] Based on the preset distortion correction algorithm and the product data of the powered-on camera, distortion calibration is performed to obtain distortion parameters;

[0025] Based on the powered-on camera and the distortion parameters, after updating the pre-calibrated association between the camera and the distortion parameters, image information collected by multiple cameras is obtained.

[0026] Based on the aforementioned technical methods, each time a camera is powered on, it is checked whether the camera has been calibrated. If it has been calibrated, then the camera is calibrated. In some applications, this method can increase the flexibility of camera combination replacement.

[0027] Retrieve product data for the camera from the preset storage unit, including:

[0028] Access the corresponding storage unit based on the mapping relationship between the camera's image sensor and the storage unit;

[0029] Access the camera information base address in the storage unit, and obtain the basic information, lens information, camera intrinsic parameters and camera extrinsic parameters of the corresponding camera based on the base address;

[0030] Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera supplier information, software and hardware number and version number and configurable parameters based on the offset address. The configurable parameters include video output pixels, video frame rate and data format.

[0031] Based on the aforementioned technical means, when a 360-degree surround view module is assembled using cameras from multiple suppliers, the product data corresponding to the cameras can be retrieved, and the initial images collected by different cameras can be processed to obtain a target image that meets the stitching requirements.

[0032] The method for obtaining configurable parameters based on the offset address:

[0033] Based on the offset address, obtain the configurable parameters corresponding to the preset specifications. All cameras have the same preset specifications.

[0034] Using the aforementioned technical methods, the configurable parameters of each camera can be configured consistently, ensuring display quality.

[0035] A panoramic image processing device, the device comprising:

[0036] The acquisition module is configured to acquire target images captured by multiple cameras, each with different distortion parameters.

[0037] The determination module is configured to determine the distortion parameters corresponding to each camera from a pre-calibrated association between cameras and distortion parameters;

[0038] The distortion correction and fusion module is configured to correct the distortion of the target image acquired by the camera according to the distortion parameters of the camera, and then fuse all the corrected images to obtain a panoramic image.

[0039] An electronic device includes: a memory and a processor;

[0040] The memory is configured to store computer instructions; the processor is configured to implement the method described above according to the computer instructions stored in the memory.

[0041] Optionally, the electronic device includes a vehicle.

[0042] A computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the method described above.

[0043] A computer program product comprising computer instructions which, when executed by a processor, are used to implement the method described above.

[0044] The beneficial effects of the embodiments disclosed herein are as follows:

[0045] (1) Product data corresponding to cameras provided by various suppliers are stored in the storage unit in advance. When calibrating the installed cameras, the product data corresponding to different cameras can be retrieved to calibrate the cameras. After the cameras are calibrated, when processing the images collected by different cameras during power-on use, the distortion parameters, camera intrinsic parameters, camera extrinsic parameters and other information corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and the pre-written product data. The images collected by each camera are converted, distorted, and stitched to obtain panoramic images. In the above way, cameras provided by different suppliers can be mixed and assembled to form a 360-degree surround view module.

[0046] (2) In this embodiment of the present disclosure, before distortion correction and stitching, the images are converted into the same bowl-shaped model based on the product data of the camera. Then, during distortion correction, an SDK algorithm is used to call the distortion parameters corresponding to the camera to perform distortion correction on the target image acquired by the camera. This can realize distortion correction and stitching processing on images acquired by cameras with different schemes. It is not necessary to use a different SDK algorithm package for each camera to perform distortion correction, thus reducing the deployment of SDK algorithms.

[0047] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0048] Figure 1 is a schematic diagram of an application scenario of the panoramic image processing method provided in the embodiment of this application;

[0049] Figure 2 is a flowchart illustrating the panoramic image processing method provided in an embodiment of this application;

[0050] Figure 3 is a detailed flowchart of step S201 in Figure 2;

[0051] Figure 4 is a flowchart illustrating the panoramic image processing method provided in an embodiment of this application;

[0052] Figure 5 is a schematic diagram of the storage method of product data of the camera provided in the embodiment of this application. Embodiments of the present invention

[0053] In implementing the embodiments of this disclosure, it was found that current image processing methods (such as distortion correction algorithms and stitching algorithms) are incompatible with different types of cameras, meaning that the four cameras must be of the same type (e.g., have the same parameters). Therefore, surround-view camera devices all use cameras from the same supplier for combination, and cannot be mixed based on different cameras, resulting in poor compatibility of image processing methods.

[0054] The embodiments of this disclosure will be described below with reference to the accompanying drawings and examples. Those skilled in the art can easily understand other advantages and effects of the embodiments of this disclosure from the content disclosed in this specification. The embodiments of this disclosure can also be implemented or applied through other different embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the embodiments of this disclosure. It should be understood that the embodiments are only for illustrating technical solutions and are not intended to limit the scope of protection of the embodiments of this disclosure.

[0055] The illustrations provided in the following embodiments are only schematic representations of the embodiments of this disclosure. Therefore, the drawings only show the components related to the embodiments of this disclosure and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0056] Panoramic imaging refers to capturing a wide field of view, typically 360 degrees, using camera equipment. This allows viewers to see the entire surrounding environment in a single image or video. Panoramic imaging is usually based on a camera system using four cameras simultaneously capturing images from different directions, which are then stitched together to create a complete panoramic view. It has applications in automotive, virtual reality (VR), security monitoring, and video conferencing. Its widespread use in the automotive industry is particularly noteworthy, as panoramic imaging (360-degree surround-view cameras) provides a more comprehensive field of view, helping drivers better understand their surroundings, avoid collisions, and enhance driving safety.

[0057] Currently, 360° surround view camera systems mainly consist of several components, including a lens, sensor (image sensor), ISP (Image Signal Processor), power chip, storage chip, and SERDES (video transmission chip). Generally, multiple cameras in the same product are procured together and installed from the same supplier. Therefore, current image processing methods for 360° surround view camera systems (such as distortion correction algorithms and stitching algorithms) are adapted to cameras from the same supplier and are not compatible with different types of cameras (cameras from different suppliers). In other words, it requires all four cameras to be of the same type (e.g., identical camera parameters). Therefore, surround view camera systems use cameras from the same supplier in combination; mixing different cameras is not possible due to poor compatibility of image processing methods.

[0058] In some special products, cameras from different suppliers are installed on the same product. However, this usually involves calling different SDK algorithm packages to process the images captured by the cameras from different suppliers. This method requires high computing power.

[0059] Based on this, this application provides a panoramic image processing method. In scenarios where cameras from different suppliers are combined and mixed, an SDK algorithm package can be used to perform distortion correction and stitching processing on images acquired by the mixed cameras.

[0060] The following uses a 360-degree surround view module for vehicles as an example to illustrate various implementation examples of the panoramic image processing method. The application of the panoramic image processing method proposed in this application includes, but is not limited to, 360-degree surround view modules for vehicles.

[0061] Figure 1 is a schematic diagram of a scenario provided in an embodiment of this application. As shown in Figure 1, the 360-degree surround view module of the vehicle includes at least four cameras 10, which are distributed in front, rear, left and right of the vehicle body, respectively. The perspectives of the four cameras 10 are combined to form a 360-degree surround view of the vehicle.

[0062] In this embodiment, the camera 10 includes a lens, an image sensor, an image signal processor (ISP), a power chip, a storage chip, and a video transmission chip (SERDES). The ISP can be integrated into the sensor or exist as a separate chip within the camera assembly.

[0063] The system comprises a lens configured to capture light and focus it onto an image sensor. The image sensor is configured to convert light signals into electrical signals, generating raw digital image data. Each lens corresponds to one sensor. An image signal processor (ISP) is configured to perform preliminary processing on the raw image data generated by the sensor, including noise reduction, white balance adjustment, color correction, gamma correction, and sharpening. This ensures consistent image quality during subsequent processing and stitching. A video transmission chip is configured to transmit the processed image data to a central processing unit (CPU) or an image processing unit (GPU). Transmission can be accomplished via various interfaces, such as MIPI, USB, and Ethernet.

[0064] In this embodiment, the four cameras can be provided by different suppliers, meaning their distortion parameters can differ. However, in vehicle applications, the video transmission chip and the video decoding chip on the multimedia host side need to be paired. Since the video decoding chip for panoramic imaging is typically an integrated four-in-one solution—one chip decodes the image signals from four panoramic cameras—it is required that the video transmission chips for the four cameras be consistent. Furthermore, the choice of video transmission chip should only affect the underlying driver configuration software, not the panoramic imaging SDK algorithm package. This allows for the use of a single SDK algorithm package to process images captured by cameras from different suppliers. Therefore, using the same model or manufacturer of video transmission chips from different suppliers allows for mixed camera installations.

[0065] The power chip provides power to the entire camera module and automatically matches the voltage of each chip. It only affects the power-on sequence of the camera and does not affect the SDK algorithm package software. Therefore, the type of power chip in cameras from different suppliers has no impact on the image processing process.

[0066] The horizontal field of view (HFOV), vertical field of view (VFOV), distortion coefficient, intrinsic and extrinsic parameters of the lens, as well as the pixel count and pixel size of the sensor chip, have a significant impact on image distortion correction and 360-degree surround view stitching algorithms, and also directly affect the panoramic image SDK algorithm package. Therefore, the panoramic audio SDK algorithm package's compatibility with cameras from different vendors is actually a compatibility adaptation of different combinations of the two key components: the lens and the sensor.

[0067] Surround-view camera sensors typically come in 1-megapixel and 3-megapixel resolutions. When four similar cameras are mixed and matched on the same vehicle, the significant pixel differences can lead to large variations in the clarity of the four angles after stitching the surround view, negatively impacting the user experience. Therefore, when selecting cameras from different suppliers, it's advisable to choose sensors with the same pixel specifications. Based on this, cameras from different suppliers can then be mixed and matched on the same vehicle.

[0068] The image processing procedure will be explained in detail below.

[0069] Figure 2 is a flowchart illustrating the panoramic image processing method provided in an embodiment of this application. As shown in Figure 2, the method includes the following steps:

[0070] S201, acquire target images obtained by multiple cameras respectively, with different distortion parameters corresponding to the multiple cameras;

[0071] First, pre-write product data for cameras from multiple suppliers in the FLASH memory. This product data includes at least one of the camera's basic information, lens information, internal camera parameters, and external camera parameters. Then, establish a relationship between the camera and the product data storage unit to facilitate the retrieval of the corresponding product data for distortion calibration and / or image conversion processing.

[0072] The basic information of a camera includes the image sensor model and firmware version, camera supplier information, hardware and software serial numbers and version numbers, video output pixels, video frame rate, and data format. The camera lens information includes the lens model, lens horizontal / vertical angle, and lens distortion parameters.

[0073] Then, after the cameras are initially installed in the vehicle, distortion calibration is performed. Using pre-written product data for each supplier's cameras, the distortion coefficients of the installed cameras are retrieved. A preset calibration method (e.g., the checkerboard method) is used to calibrate the installed cameras, obtaining their corresponding distortion parameters. Since the product data for each camera on the vehicle differs, calibration is performed separately based on the product data for each camera, obtaining the distortion parameters for each camera. The distortion parameters and product data are then correlated to form a relationship between the cameras and the distortion parameters. Optionally, this relationship can be presented in a table format.

[0074] After the cameras are calibrated, power can be supplied to them. In this embodiment, a POC power supply method is used. After the vehicle-side multimedia host powers on the four surround-view cameras, communication between the vehicle-side multimedia host and the cameras is established via I2C signals. The underlying driver identifies the sensor model by its ID through polling, loads the corresponding firmware, and powers on the camera.

[0075] After the cameras are powered on, they begin acquiring image information. The video transmission chip transmits the images to the image processor, which then obtains the initial images captured by multiple cameras. After obtaining the initial images from each camera, the image processor performs coordinate transformation on the initial images to obtain the target image captured by multiple cameras. Subsequently, distortion correction and fusion processing are performed on the target image.

[0076] As an example, since the initial image output by each camera is in its own camera coordinate system, and the initial images output by different cameras are in different coordinate systems, it is impossible to perform distortion correction and stitching fusion on multiple images. Therefore, coordinate transformation processing can be performed on the initial images captured by the cameras to obtain target images in the same coordinate system.

[0077] Optionally, as shown in Figure 3, taking a single camera as an example, the initial image captured by the camera can be processed to obtain the target image in the following ways:

[0078] S301 retrieves the camera's product data from the preset storage unit and determines the transformation matrix for converting the original image captured by the camera from camera coordinates to world coordinates based on the camera's product data.

[0079] For example, based on the camera's intrinsic and extrinsic parameters, the transformation matrix corresponding to the transformation of the original image captured by the camera from camera coordinates to world coordinates can be determined. Different cameras have different intrinsic and extrinsic parameters, resulting in different transformation matrices. Therefore, the transformation matrix can be calculated by retrieving the corresponding intrinsic and extrinsic parameters for each camera.

[0080] As an example, as shown in Figure 5, product data for cameras from multiple suppliers is pre-written into the FLASH memory. To improve the compatibility of the application layer software, a base address + offset address method is used to write the product data for each supplier's camera. That is, the required information is fixed within a certain address range, and the order remains consistent. The base address is provided by the underlying driver or bound to the sensor model, ensuring that accessing the same location specified by the offset address of different solutions yields consistent information.

[0081] Therefore, the electronic device in this application embodiment can obtain the camera's product data in the following ways:

[0082] Access the corresponding storage unit based on the mapping relationship between the camera's image sensor and the storage unit;

[0083] Access the camera information base address in the storage unit, and obtain the basic information, lens information, camera intrinsic parameters and camera extrinsic parameters of the corresponding camera based on the base address;

[0084] Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera supplier information, software and hardware number and version number and configurable parameters based on the offset address. The configurable parameters include video output pixels, video frame rate and data format.

[0085] When it is necessary to obtain the camera's intrinsic and extrinsic parameters, the corresponding stored camera intrinsic and extrinsic parameters can be retrieved through the base address of the camera information. Then, using the camera's intrinsic and extrinsic parameters, the transformation matrix corresponding to the camera's transformation to world coordinates can be calculated.

[0086] When it is necessary to obtain basic information about the camera, the corresponding basic information is obtained through the camera information offset address, including the image sensor model and firmware version, camera supplier information, software and hardware numbers and version numbers, and configurable parameters.

[0087] In this embodiment, configurable parameters refer to parameters with multiple sets of selectable configurations, such as multiple sets of video output pixels, multiple sets of video frame rates, and multiple data formats. In this embodiment, when obtaining configurable parameters, configurable parameters are selected based on preset specifications, and all cameras obtain configurable parameters based on preset specifications. In this way, the configurable parameters of all cameras can be kept consistent, thereby ensuring display quality in applications with mixed camera setups.

[0088] In this embodiment, each type of camera's product data corresponds to a storage unit. The relationship between the camera's image sensor model and the storage unit is established; the uniqueness of the image sensor model allows the electronic device to locate accurate product data. Furthermore, in this embodiment, the same offset address is used for parameters of the same type, allowing for accurate retrieval of product data corresponding to each camera through the offset address.

[0089] With the above settings, when a 360-degree surround view module is assembled based on cameras from multiple suppliers, the product data corresponding to the cameras can be retrieved, and the initial images collected by different cameras can be processed to obtain the target image that meets the stitching requirements.

[0090] S302, In the world coordinate system, establish a bowl-shaped model with the target point as the origin.

[0091] In the bowl-shaped model, the initial images captured by each camera in the world coordinate system can be transformed to the world coordinate system using their corresponding transformation matrix. Furthermore, by combining the bowl-shaped model with the initial images from each camera, the initial images from each camera can be transformed into the bowl-shaped model, thus presenting the target images corresponding to each camera within the bowl-shaped model.

[0092] Taking vehicle applications as an example, world coordinates refer to the vehicle's world coordinates (vehicle coordinate system).

[0093] A bowl-shaped model refers to a model that simulates a real-world three-dimensional curved surface, specifically a 3D bowl-shaped surface, onto which images captured by a camera (such as a fisheye camera) are projected. The 3D bowl-shaped surface is composed of multiple small facets, and by calculating the correspondence between these facets and the actual image, pixels from the image are mapped onto the 3D bowl-shaped surface.

[0094] Bowl-shaped model creation is suitable for generating surround-view images for autonomous driving systems, such as in Automated Parking Assist (AVM) systems, which enhances the realism and three-dimensionality of the visual experience by presenting the scene around the vehicle to the driver in three dimensions.

[0095] S303, Based on the transformation matrix, the original image is mapped onto the bowl-shaped model in the world coordinates to obtain the target image.

[0096] It is understandable that the pixel coordinates of the original image are mapped to the bowl-shaped model based on the coordinate transformation matrix, thereby transforming the initial image into the bowl-shaped model.

[0097] As an example, the target point can be the center of the vehicle body, and four cameras are arranged along the four directions of the center of the vehicle body. Then, a bowl-shaped model is built with the center of the vehicle body as the origin. The original images of the cameras are mapped onto the bowl-shaped model. The converted image can evenly present the images in the four directions, making the image stitching effect better.

[0098] Multiple cameras obtain their respective transformation matrices using the methods described above. Then, based on these transformation matrices, the original images are mapped onto the bowl-shaped model, thus forming a bowl-shaped image composed of multiple target images within the bowl-shaped model.

[0099] In other words, as long as different cameras access the correct lens distortion table and intrinsic / extrinsic parameters during the above process, the stitching will be normal when projected onto world coordinates, thus not affecting the stitching effect.

[0100] In this embodiment, all original images are converted to the same bowl-shaped model. On the one hand, images from different suppliers can be converted to the same coordinate system. On the other hand, the target image is displayed in a bowl-shaped model, making the target image closer to the image captured by the surround-view camera (or fisheye camera), resulting in more accurate image distortion correction and better image display effect after stitching.

[0101] S202, Determine the distortion parameters corresponding to each camera from the pre-calibrated correlation between cameras and distortion parameters;

[0102] Since the distortion parameters corresponding to multiple cameras are different, when performing distortion correction and fusion processing on target images acquired by multiple cameras respectively, different distortion parameters can be called to perform distortion correction on the target images.

[0103] In this embodiment, the calibration of the cameras is based on the product data of each camera, forming a distortion table for each camera (the correlation between the camera and the distortion parameter). When determining the camera on the vehicle, the corresponding distortion parameter of the camera can be retrieved from the pre-calibrated correlation between the camera and the distortion parameter based on the camera's product data.

[0104] In this embodiment, steps S201 and S202 are not limited by the described order of actions; steps S201 and S202 can be performed in other orders or simultaneously.

[0105] S203, based on the distortion parameters corresponding to the camera, performs distortion correction on the target image acquired by the camera, and fuses all the corrected images to obtain a panoramic image.

[0106] When the SDK algorithm package performs distortion correction on a target image from a specific camera, it retrieves the distortion parameters corresponding to that camera and then applies the distortion correction to the target image using those parameters. Furthermore, after distortion correction, multiple target images are stitched together to form a panoramic image.

[0107] Based on the above process, the following example illustrates the process:

[0108] Taking a vehicle equipped with four cameras from different suppliers as an example, let's say camera n 11 Camera n 12 Camera n 21 Camera n 22 Among them, camera n 11 and camera n 12 Camera n is a camera supplied by a supplier. 11 and camera n 12 The product data is the same; camera n 21 and camera n 22 Camera n, supplied by another supplier 21 and camera n 22 The product data is the same. However, the camera n 11 Camera n 12 With camera n 21 Camera n 22 The product data differs. The vehicle's memory consists of multiple storage units, each storing at least one set of product data for the supplier's cameras. The image sensor model from the product data is associated with the storage unit, using the image sensor model as the lookup index. Assume there are 6 sets of product data, designated as product data groups M1 to M6.

[0109] The camera installed on the vehicle is calibrated to calibrate camera n. 11 Calibration, for example: via camera n 11 To find the image sensor model, you can locate the associated storage unit. For example, by comparing the image sensor model with a lookup index, you can find the camera n in the storage unit associated with the same image sensor model. 11 Product data M1. Then, based on the preset calibration template and camera n... 11 Product data M1, for camera n 11 Perform distortion calibration to obtain the image from camera n. 11 The distortion parameter X1 is used to establish the camera n 11- The correlation of distortion parameter X1. It is worth noting that the product data of the camera includes lens information, which includes the horizontal and vertical angles of the lens. When the horizontal or vertical angle of the lens changes, the distortion of the captured image will be different. Based on this, the lens can be calibrated at different horizontal or vertical angles. Therefore, the correlation between the camera and the distortion parameter, including the correlation between lens information and distortion parameters, means that one camera can correspond to multiple sets of distortion parameters, and each set of distortion parameters corresponds to different lens information.

[0110] Using the above method, camera n is sequentially... 12 Camera n 21 Camera n 22 Calibration is performed. Among them, camera n... 11 and camera n 12 Since the cameras are from the same supplier, their corresponding product data is all M1, but because camera n 11 and camera n 12 The installation location differs, therefore the lens information differs, and the corresponding distortion parameters differ as well. For example, camera n 11 -Establish a correlation between distortion parameter X1 and camera n 12 -Establish a correlation using distortion parameter X1. (Camera n) 21 and camera n 22 The cameras are from the same supplier, so there are n cameras. 21 and camera n 22 Even if the product data is the same, say M2, the lens information will be different depending on the installation location, and thus the distortion parameters obtained from calibration will also be different.

[0111] The four cameras begin capturing images after being powered on. (The last sentence appears to be incomplete and possibly refers to a separate task involving cameras n.) 11 Taking the processing of acquired images as an example, the image processing process is illustrated as follows: The processor acquires data from camera n... 11 The initial image transmitted, and viewed through camera n 11 Image sensor model, retrieve camera n 11 Product data M1. Calculate camera n using the camera intrinsic and extrinsic parameters in M1. 11 The transformation matrix is ​​used to convert camera coordinates to world coordinates, and a bowl-shaped model is created in world coordinates. Then, the camera n... 11 The initial image transmitted is mapped onto the bowl-shaped model based on the transformation matrix to obtain target image 1. The other three cameras are similarly mapped onto the bowl-shaped model sequentially to obtain target images 2, 3, and 4. Here, since there are n cameras... 11 Camera n 12With camera n 21 Camera n 22 The product data is different, so the camera n 11 Camera n 12 With camera n 21 Camera n 22 Since the corresponding transformation matrices are different, when processing the initial image, the corresponding product data can be called to calculate the transformation matrix.

[0112] After generating four target images in the bowl-shaped model, distortion correction is performed on each of the four target images, and then the distortion-corrected target images are fused and stitched together. This is done for camera n. 11 Taking the distortion correction of the acquired target image 1 as an example: through camera n 11 Image sensor signals, locate camera n 11 The SDK algorithm package uses the corresponding distortion parameter X1 to perform distortion correction processing on target image 1. Similarly, the distortion correction processing of the target images acquired by the other three cameras is performed in the same way as described above.

[0113] In this embodiment, the target image is stitched together by fusing features based on overlapping positions.

[0114] In the assembly process of a vehicle's 360-degree surround view module, cameras from multiple suppliers may be present. If all four cameras in a 360-degree surround view module installed on the same vehicle are from the same supplier, the cameras can be distinguished before installation. However, this scenario is prone to installation errors, leading to the 360-degree surround view module malfunctioning. In this embodiment, cameras from different suppliers can be installed on the same vehicle to form a 360-degree surround view module. In this scenario, installers can use different cameras for assembly without distinguishing them, improving installation efficiency and preventing the mixing of cameras from affecting the normal operation of the 360-degree surround view module.

[0115] In this embodiment, product data corresponding to cameras from various suppliers are pre-stored in the storage unit. When calibrating the installed cameras, the product data corresponding to different cameras can be retrieved for calibration. After camera calibration, during power-on use, when processing images captured by different cameras, the distortion parameters, camera intrinsic parameters, and camera extrinsic parameters corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and pre-written product data. The images captured by each camera are then converted, distortion corrected, and stitched together to obtain a panoramic image. In this way, a 360-degree surround view module can be formed by mixing cameras from different suppliers.

[0116] Figure 4 is a schematic flowchart of the panoramic image processing method provided in this application embodiment. This embodiment is based on the above embodiment. Since this embodiment is applicable to mixed installations of cameras from different suppliers, the cameras may be replaced when they malfunction or based on user needs. To avoid the inability to process images acquired by the new camera in scenarios where the camera is replaced due to malfunction or changes in user needs, this application embodiment solves this problem in the following way, such as the method including:

[0117] S401, when the camera is detected to be powered on, the pre-calibrated camera is queried. The pre-calibrated camera is associated with distortion parameters and stored in the storage unit.

[0118] When the vehicle starts, the camera is powered on, and this power-on status is detected.

[0119] In one example, when a camera is powered on, it is queried whether the powered-on camera has been pre-calibrated. This example determines whether the powered-on camera needs calibration by comparing it with pre-calibrated cameras in the system.

[0120] For pre-calibrated cameras, their image sensor models are associated with distortion parameters. You can look up the image sensor model of each camera in the camera-distortion parameter table to find the pre-calibrated camera.

[0121] S402, determine whether the pre-calibrated cameras include powered-on cameras;

[0122] By comparing the image sensor signal of the powered-on camera with the image sensor model of the pre-calibrated camera, it can be determined whether the pre-calibrated camera includes the powered-on camera.

[0123] If it is determined that the pre-calibrated cameras include powered-on cameras, that is, the powered-on cameras are calibrated, then S403 is executed to acquire the target image captured by the powered-on cameras. If multiple cameras are powered on, target images are acquired by each of the multiple cameras respectively. Then, the following steps S407 and S408 are executed to process the target images acquired by each powered-on camera.

[0124] In other words, if the powered-on camera has already been calibrated, the product parameters corresponding to the powered-on camera can be directly called to process the images captured by the powered-on camera.

[0125] If it is determined that the pre-calibrated cameras do not include the powered-on cameras, that is, the powered-on cameras have not been calibrated, then S404 is executed to query the product data corresponding to the powered-on cameras from the preset storage unit. The product data includes at least one of the camera's basic information, lens information, camera internal parameters, and camera external parameters.

[0126] If the powered-on camera is not one of the four previously calibrated cameras, the system retrieves the corresponding product data, such as distortion coefficient, intrinsic parameters, and extrinsic parameters, from the preset storage unit to calibrate the powered-on camera.

[0127] S405 performs distortion calibration and obtains distortion parameters based on the preset distortion correction algorithm and the product data of the powered-on camera;

[0128] In this embodiment, a calibration method is used for all four cameras. When calibrating each camera, its own distortion coefficient is used to obtain the corresponding distortion parameters.

[0129] This embodiment takes fisheye camera calibration as an example: Fisheye cameras suffer from barrel distortion, so a fisheye model is selected as the imaging model, and the checkerboard calibration method is used to calibrate the camera, as follows:

[0130] A black and white checkerboard pattern was created as a template for image information calibration. The calibration template was observed from different angles to obtain image information from different perspectives. Feature information was extracted from the checkerboard template. During the acquisition of template image information, both the camera and the checkerboard calibration board could be moved freely. Finally, based on the fisheye model and the extracted feature point information, distortion model parameters were calculated, thereby achieving distortion correction and calibration of the camera.

[0131] S406, based on the product data and distortion parameters of the powered-on cameras, updates the pre-calibrated association between the cameras and distortion parameters, and then acquires image information collected by multiple cameras respectively.

[0132] After obtaining the distortion parameters of the powered-on camera, the pre-calibrated association between the camera and the distortion parameters is updated. In this way, the distortion parameters corresponding to the camera are stored in the storage unit, which makes it convenient to call the distortion parameters corresponding to the camera during image processing.

[0133] After the cameras are calibrated, the following steps S407 and S408 are performed to process the target images acquired by each powered-on camera:

[0134] S407 determines the distortion parameters corresponding to each camera from the pre-calibrated correlation between cameras and distortion parameters;

[0135] S408 performs distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and fuses all the corrected images to obtain a panoramic image.

[0136] In this embodiment, steps S407 and S408 are similar to S202 and S203 in the embodiment shown in FIG2 above, and can be referred to the above embodiment.

[0137] In this embodiment, the camera can be replaced as needed without affecting the normal use of the 360-degree surround view module, demonstrating strong compatibility.

[0138] Optionally, in some examples, if a user wants to replace a camera that is not among the pre-written supplier's cameras, a replacement request can be initiated through the after-sales service. The after-sales service receives the replacement request, determines the product data of the camera to be replaced based on the request, and writes the product data of the camera to be replaced into the vehicle's infotainment system's storage unit. After the vehicle's infotainment system's storage unit is updated, the user can replace the camera. This method achieves compatibility with the camera to be replaced. Therefore, in this example, the flexibility of the 360-degree surround view module's combination method is improved.

[0139] This application embodiment also provides a panoramic image processing device, the device comprising:

[0140] The acquisition module is configured to acquire target images captured by multiple cameras, each with different distortion parameters.

[0141] The determination module is configured to determine the distortion parameters corresponding to each camera from a pre-calibrated association between cameras and distortion parameters;

[0142] The distortion correction and fusion module is configured to correct the distortion of the target image acquired by the camera according to the distortion parameters of the camera, and then fuse all the corrected images to obtain a panoramic image.

[0143] The panoramic image processing device provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0144] This application also provides an electronic device, which includes: a memory and a processor;

[0145] The memory is configured to store computer instructions; the processor is configured to execute the computer instructions stored in the memory to implement the methods involved in the various embodiments above.

[0146] The electronic device also includes a communication interface and a CAN bus. The processor is configured to provide computing and control capabilities and can be a GPU, CPU, NPU, MCU, FPGA, etc. Storage devices include internal memory and non-volatile memory. The non-volatile memory stores the computer instructions that implement the above methods. The internal memory provides the environment for program startup and execution. The communication interface is configured for wired or wireless communication with external terminals.

[0147] In this embodiment, the electronic device can be an electronic component in a car, such as a vehicle infotainment system, or it can be the car itself.

[0148] Taking a car as an example, a car includes a multimedia host, connecting wiring harnesses, and four surround-view cameras with various configurations. Among them, the surround-view cameras are module solutions composed of different combinations of key components such as lenses and sensors.

[0149] The multimedia host and the camera are physically connected via LVDS, and GMSL is used for communication to transmit camera video signals. At the same time, I2C communication is also based on LVDS. It is also a processing module that integrates SDK algorithm packages such as panoramic image distortion correction and stitching.

[0150] Given the wide variety of fisheye lens models and the increasing number of sensor options due to technological advancements, a multimedia host is used to read the solution information of the connected cameras and perform corresponding distortion correction and stitching processing to display the 360° panoramic image on the central control screen. This improves the compatibility of the panoramic image algorithm, enables the mixing and matching of multiple solution modules in one vehicle, and reduces the risk of hardware supply issues.

[0151] This disclosure also provides a computer-readable storage medium / computer program product, which stores computer control instructions / computer program product including computer control instructions, which, when executed by a processor, are used to implement the methods involved in the above embodiments.

[0152] The above embodiments are merely illustrative of the technical solutions, and the protection scope of this disclosure is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the embodiments of this disclosure are all within the protection scope of this disclosure.

[0153] The various modules, sub-modules, units, and sub-units mentioned in the above technical description do not refer solely to functional divisions or virtual components at the pure software logic level. Rather, they can be implemented as physical devices, components, parts, or combinations thereof with specific hardware or physical structures. They can also be implemented using a combination of hardware and software, i.e., hardware executes corresponding software instructions to achieve the corresponding function. The aforementioned modules, sub-modules, units, and sub-units can be exemplified as Application-Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), Programmable Logic Devices (PLDs), Discrete Logic Circuits, Central Processing Units (CPUs), Microprocessors, Microcontrollers, Microcontrollers, Digital Signal Processors (DSPs), Dedicated Function Chips, Storage Devices, Interface Circuits, and other hardware entities or hardware component combinations with functions such as data processing, signal transmission, and logical operations. Furthermore, the aforementioned hardware entities can be equipped with corresponding software programs, firmware, or instruction sets to form a functional carrier that combines hardware foundation and software execution capabilities.

Claims

1. A panoramic image processing method, comprising: Acquire target images captured by multiple cameras, each with different distortion parameters; The distortion parameters corresponding to each camera are determined from the pre-calibrated correlation between cameras and distortion parameters. The distortion of the target image acquired by the camera is corrected according to the distortion parameters of the camera, and all the corrected images are fused together to obtain a panoramic image.

2. The method according to claim 1, wherein, The acquisition of target images obtained from multiple cameras includes: The product data of the camera is retrieved from the preset storage unit, and the transformation matrix of the original image captured by the camera is determined from the camera coordinate system to the world coordinate system based on the product data of the camera. In the world coordinate system, a bowl-shaped model is established with the target point as the origin; Based on the transformation matrix, the original image is mapped onto the bowl-shaped model in the world coordinates to obtain the target image. The original images from multiple cameras are all mapped onto the bowl-shaped model to obtain multiple target images.

3. The method according to claim 1, wherein, The acquisition of target images obtained from multiple cameras includes: When a camera is detected to be powered on, the pre-calibrated cameras are queried. The pre-calibrated cameras are associated with distortion parameters and stored in the storage unit. When the pre-calibrated cameras include powered-on cameras, the target image captured by the powered-on cameras is obtained. When multiple cameras are powered on, the target images captured by each of the multiple cameras are obtained respectively.

4. The method according to claim 3, wherein, The step of acquiring target images captured by multiple cameras also includes: When it is determined that the pre-calibrated cameras do not include powered-on cameras, the product data corresponding to the powered-on cameras is retrieved from the preset storage unit. The product data includes at least one of the camera's basic information, lens information, camera internal parameters, and camera external parameters. Based on the preset distortion correction algorithm and the product data of the powered-on camera, distortion calibration is performed to obtain distortion parameters; Based on the powered-on camera and the distortion parameters, after updating the pre-calibrated association between the camera and the distortion parameters, image information collected by multiple cameras is obtained.

5. The method according to any one of claims 2 to 4, wherein, Retrieve product data for the camera from the preset storage unit, including: Access the corresponding storage unit based on the mapping relationship between the camera's image sensor and the storage unit; Access the camera information base address in the storage unit, and obtain the basic information, lens information, camera intrinsic parameters and camera extrinsic parameters of the corresponding camera based on the base address; Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera supplier information, software and hardware number and version number and configurable parameters based on the offset address. The configurable parameters include video output pixels, video frame rate and data format.

6. The panoramic image processing method according to claim 5, wherein, The method for obtaining configurable parameters based on the offset address: Based on the offset address, obtain the configurable parameters corresponding to the preset specifications. All cameras have the same preset specifications.

7. A panoramic image processing device, comprising: The acquisition module is configured to acquire target images captured by multiple cameras, each with different distortion parameters. The determination module is configured to determine the distortion parameters corresponding to each camera from a pre-calibrated association between cameras and distortion parameters; The distortion correction and fusion module is configured to correct the distortion of the target image acquired by the camera according to the distortion parameters of the camera, and then fuse all the corrected images to obtain a panoramic image.

8. An electronic device, comprising: Memory, processor; The memory is configured to store computer instructions; The processor is configured to implement the method as described in any one of claims 1 to 6, according to computer instructions stored in the memory.

9. The electronic device according to claim 8, wherein, The electronic devices include automobiles.

10. A computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the method as claimed in any one of claims 1 to 6.

11. A computer program product comprising computer instructions that, when executed by a processor, are used to implement the method as claimed in any one of claims 1 to 6.

Images