Camera calibration evaluation method and apparatus, electronic device and storage medium

Abstract

The present application provides a camera calibration evaluation method and apparatus, an electronic device and a storage medium. The camera calibration evaluation method comprises: acquiring first intermediate images corresponding to respective cameras on a target device; determining a mapping relationship between the first intermediate images and virtual projected images of virtual cameras; on the basis of the mapping relationship, mapping the first intermediate images into the virtual projected images to obtain second intermediate images corresponding to the respective cameras; splicing the second intermediate images to obtain spliced projected images; and, on the basis of the spliced projected images, evaluating respective calibration extrinsic parameters of the cameras. The present application is used for solving the problem of there being no way to quickly and intuitively evaluate calibration extrinsic parameters of multiple cameras.

Description

Camera calibration and evaluation methods, devices, electronic equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411552673.8, filed on November 1, 2024, entitled "Camera Calibration Evaluation Method, Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of camera calibration technology, and in particular to a camera calibration evaluation method, apparatus, electronic device and storage medium. Background Technology

[0003] Camera calibration is a crucial step in the production process of autonomous vehicles, mobile robots, and other machine vision products, playing a core role in ensuring the accurate operation of the vision system. Camera calibration lays a solid foundation for subsequent tasks such as 3D reconstruction, object detection, and path planning. Especially in complex autonomous driving scenarios, high-precision camera calibration can significantly improve the system's environmental perception capabilities, thereby enhancing the navigation accuracy and safety of vehicles or robots.

[0004] However, despite the undeniable importance of camera calibration technology for machine vision products, the industry still faces significant challenges in handling multi-camera systems, particularly in evaluating multi-camera calibration extrinsic parameters. Multi-camera systems often require extremely high precision in the spatial relationships between cameras to achieve seamless stitching of the surrounding environment and accurate acquisition of depth information. However, most existing calibration methods focus on single-camera systems. For systems containing multiple cameras, there are currently no effective methods for evaluating multi-camera calibration extrinsic parameters, making it difficult to quickly and intuitively assess these parameters. Summary of the Invention

[0005] This application provides a camera calibration and evaluation method, apparatus, electronic device, and storage medium. The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0006] According to a first aspect of the embodiments of this application, a camera calibration and evaluation method is provided, comprising:

[0007] Acquire the first intermediate image corresponding to each camera on the target device;

[0008] Determine the mapping relationship between the first intermediate image corresponding to each of the cameras and the virtual projection image of the virtual camera;

[0009] Based on the mapping relationship, the first intermediate image corresponding to each of the cameras is mapped onto the virtual projection image to obtain the second intermediate image corresponding to each of the cameras;

[0010] The second intermediate images corresponding to each of the cameras are stitched together to obtain the stitched projection image.

[0011] Based on the stitched projected image, the calibration extrinsic parameters of each of the cameras are evaluated.

[0012] Optionally, determining the mapping relationship between the first intermediate image projected onto the virtual projection image of the virtual camera corresponding to each of the cameras includes:

[0013] Perform the following operations on any preset pixel in the virtual projected image of the virtual camera:

[0014] Based on the first two-dimensional coordinates of the preset pixel, determine the first projection light path that is projected onto the preset pixel by the virtual camera;

[0015] Determine the first three-dimensional coordinates of the target three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera;

[0016] Based on the calibration extrinsic parameters of the target camera and the first three-dimensional coordinates, the second three-dimensional coordinates of the target space three-dimensional point in the three-dimensional coordinate system of the target camera are determined, wherein the target camera is at least one camera among the cameras that corresponds to the target space three-dimensional point;

[0017] Based on the second three-dimensional coordinates, a second projection optical path is determined for the projection of the target space three-dimensional point onto the target camera;

[0018] Based on the second projection optical path, the intrinsic parameters of the target camera, and the intrinsic parameters of the virtual camera, the second two-dimensional coordinates of the actual pixel points in the virtual projection image of the virtual camera projected onto the three-dimensional point of the target space are determined.

[0019] Based on the second two-dimensional coordinates, the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera is determined.

[0020] Optionally, determining the first three-dimensional coordinates of the target spatial three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera includes:

[0021] Determine the first normalized vector of the first projection optical path in the three-dimensional coordinate system of the virtual camera;

[0022] Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera;

[0023] Based on the first normalized vector and the first transformation relationship, determine the second normalized vector of the first projected optical path in the three-dimensional coordinate system of the target device;

[0024] Based on the first transformation relationship and the second normalized vector, the scaling factor is determined;

[0025] Based on the first normalized vector and the scale factor, the first three-dimensional coordinates of the target three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera are determined.

[0026] Optionally, determining the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system based on the target camera's calibration extrinsic parameters and the first three-dimensional coordinates includes:

[0027] Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera;

[0028] Based on the calibration extrinsic parameters of the target camera, a second transformation relationship is determined from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device;

[0029] Based on the first transformation relationship, the second transformation relationship, and the first three-dimensional coordinates, the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system are determined.

[0030] Optionally, determining the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera includes:

[0031] Based on the extrinsic parameters of the virtual camera, determine the third transformation relationship from the virtual camera's three-dimensional coordinate system to the world coordinate system;

[0032] Determine the fourth transformation relationship from the three-dimensional coordinate system of the target device to the world coordinate system;

[0033] Based on the third and fourth transformation relationships, a first transformation relationship is determined from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

[0034] Optionally, before determining the second projection optical path for projecting the target space three-dimensional point onto the target camera based on the second three-dimensional coordinates, the method further includes:

[0035] Based on the second three-dimensional coordinates, it is determined that the three-dimensional point in the target space can be imaged by the target camera.

[0036] Optionally, evaluating the calibration extrinsic parameters of each camera based on the stitched projected image includes:

[0037] The calibration accuracy of each camera's calibration extrinsic parameters is evaluated based on the geometric feature relationships in the real external environment and the geometric feature relationships in the stitched projected image.

[0038] According to a second aspect of the embodiments of this application, a camera calibration and evaluation apparatus is provided, comprising:

[0039] The acquisition unit is configured to acquire the first intermediate image corresponding to each camera on the target device;

[0040] The processing unit is configured to determine the mapping relationship between the first intermediate image corresponding to each of the cameras and the virtual projection image of the virtual camera;

[0041] The mapping unit is configured to map the first intermediate image corresponding to each of the cameras onto the virtual projection image based on the mapping relationship, so as to obtain the second intermediate image corresponding to each of the cameras.

[0042] The stitching unit is configured to stitch together the second intermediate images corresponding to each of the cameras to obtain a stitched projection image.

[0043] The evaluation unit is configured to evaluate the calibration extrinsic parameters of each of the cameras based on the stitched projected image.

[0044] According to a third aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor;

[0045] The memory is connected to the processor and is configured to store a program.

[0046] The processor is configured to implement the camera calibration and evaluation method as described in the first aspect by running a program in the memory.

[0047] According to a fourth aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, it implements the camera calibration and evaluation method as described in the first aspect.

[0048] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the camera calibration and evaluation method as described in the first aspect.

[0049] In this application, a first intermediate image corresponding to each camera on the target device is obtained. The mapping relationship between the first intermediate image of each camera and the virtual projection image of the virtual camera is determined. Based on this mapping relationship, the first intermediate image of each camera is mapped onto the virtual projection image to obtain a second intermediate image corresponding to each camera. The first intermediate images of each camera are then transformed to the virtual camera's viewpoint to obtain the second intermediate image of each camera from the virtual camera's viewpoint. The second intermediate images of each camera are then stitched together to obtain a stitched projection image. Based on the stitched projection image, the calibration extrinsic parameters of each camera are evaluated. By observing the stitched projection image from the virtual camera's viewpoint, the calibration extrinsic parameters of each camera on the target device can be quickly and intuitively evaluated. This method is convenient, simple, and easy to operate, solving the problem of the inability to quickly and intuitively evaluate the calibration extrinsic parameters of multiple cameras. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 is a flowchart illustrating a camera calibration and evaluation method provided in an embodiment of this application;

[0052] Figure 2 is a schematic diagram of the positional relationship between the image coordinate system and the pixel coordinate system of a virtual projection image provided in an embodiment of this application;

[0053] Figure 3 is a schematic diagram of a three-dimensional coordinate system of a camera, an image coordinate system of the first intermediate image corresponding to the camera, and a world coordinate system provided in an embodiment of this application;

[0054] Figure 4 is a flowchart illustrating step 102 provided in an embodiment of this application;

[0055] Figure 5 is a flowchart illustrating step 402 provided in an embodiment of this application;

[0056] Figure 6 is a flowchart illustrating step 502 provided in an embodiment of this application;

[0057] Figure 7 is a flowchart illustrating step 403 provided in an embodiment of this application;

[0058] Figure 8 is a schematic diagram of a first projection optical path and a second projection optical path provided in an embodiment of this application;

[0059] Figure 9 is a schematic diagram of a stitched projection image provided in an embodiment of this application, in which the calibration accuracy of each camera's calibration extrinsic parameters is qualified.

[0060] Figure 10 is a schematic diagram of a stitched projection image provided in an embodiment of this application, in which the calibration accuracy of the calibration extrinsic parameters of each camera is not up to standard.

[0061] Figure 11 is a schematic diagram of a stitched projection image provided in another embodiment of this application where the calibration accuracy of the calibration extrinsic parameters of each camera is not up to standard.

[0062] Figure 12 is a schematic diagram of the structure of a camera calibration and evaluation device provided in an embodiment of this application;

[0063] Figure 13 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] Exemplary Implementation Environment

[0066] The camera calibration and evaluation method according to embodiments of this application can be executed by electronic devices such as terminal devices or servers. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. This method can be implemented by a processor calling computer-readable program instructions stored in memory.

[0067] In some embodiments, the camera calibration and evaluation method in this application supports online, offline, and remote methods.

[0068] The online method refers to the process where the onboard equipment of the intelligent driving vehicle establishes a connection with the intelligent driving vehicle through technologies such as Ethernet and WIFI (a wireless local area network communication technology), obtains security verification, and then the onboard equipment of the intelligent driving vehicle can directly trigger the camera calibration and evaluation method in this application.

[0069] The offline method refers to manually or automatically collecting the original images and files of intrinsic parameters, distortion parameters, and calibration extrinsic parameters of each camera when the external environment of the intelligent driving vehicle meets the preset conditions, copying the original images and files of intrinsic parameters, distortion parameters, and calibration extrinsic parameters of each camera to another electronic device that has deployed the camera calibration evaluation method of this application, and running the camera calibration evaluation method of this application.

[0070] The remote method refers to establishing a connection between electronic devices and intelligent driving vehicles through technologies such as 4G, 5G, and dedicated satellite communication. After obtaining security verification, a remote command triggers the camera calibration and evaluation method in this application, or the electronic devices automatically trigger the camera calibration and evaluation method in this application when the external environment of the intelligent driving vehicle meets preset conditions, remotely reading the stitched projected image and quickly evaluating the calibration extrinsic parameters of each camera.

[0071] Exemplary methods

[0072] Referring to Figure 1, in an exemplary embodiment, a camera calibration and evaluation method is provided. As shown in Figure 1, the process of the camera calibration and evaluation method mainly includes:

[0073] Step 101: Obtain the first intermediate image corresponding to each camera on the target device.

[0074] In the exemplary embodiment, the target device refers to a device equipped with multiple cameras. For example, the target device can be a vehicle or a mobile robot. This application uses a vehicle as an example for explanation, but it is not intended to be limiting. Specifically, the target device can be an intelligent driving vehicle.

[0075] In the exemplary embodiment, the camera model of each camera on the target device can be a pinhole camera model, a fisheye camera model, or other types of camera imaging models, and this application does not limit this.

[0076] In some embodiments, the intrinsic parameters, calibration extrinsic parameters, and distortion parameters of each camera on the target device are obtained. The intrinsic and distortion parameters of each camera are provided by the camera manufacturer or other upstream modules and are not obtained through calibration after installation; it is assumed that the intrinsic and distortion parameters of the cameras after installation are qualified. The calibration extrinsic parameters of each camera are the calibration parameters to be evaluated in this application. They are obtained through production line calibration during installation or after-sales maintenance and describe the pose relationship between the camera's three-dimensional coordinate system and the vehicle's three-dimensional coordinate system.

[0077] In some embodiments, the first intermediate image corresponding to each camera may be an image corresponding to the same moment.

[0078] In some embodiments, the first intermediate image corresponding to each camera can be the original image or an image obtained by distorting the original image.

[0079] In some embodiments, where the first intermediate image corresponding to each camera is an image after distortion correction of the original image, step 101 includes: acquiring the original image corresponding to each camera on the target device; performing distortion correction operation on the original image corresponding to each camera based on the intrinsic parameters and distortion parameters of each camera to obtain the first intermediate image after distortion correction corresponding to each camera.

[0080] In some embodiments, acquiring the original images corresponding to each camera on the target device can be achieved by acquiring the original images captured by each camera on the vehicle at the same time when the external environment of the vehicle meets preset conditions; or by extracting the original images corresponding to each camera at the same time when the external environment meets preset conditions from the videos recorded by each camera on the vehicle.

[0081] In some embodiments, the preset conditions for the external environment include the following conditions: (1) The vehicle is stationary or running smoothly without bumps or significant vibrations, and the tire pressure is basically consistent, and the air suspension is turned off (vehicles without air suspension do not need to turn off the air suspension); (2) The ground where the vehicle is located is flat and without undulations (for example, the ground must be flat within 100 meters in front of and behind the vehicle and within 15 meters to the left and right); (3) There are common and continuous straight features parallel to the ground where the vehicle is located within the imaging field of each camera on the vehicle (for example, generally, a flat road with straight lane lines or curbs is selected; according to the actual application, preferably, a straight, open, flat road with three or more lanes is selected).

[0082] In some embodiments, when acquiring or extracting the original image, the vehicle is parallel to the lane lines and within the lane.

[0083] In some embodiments, the original image can be in multiple formats such as RAW (raw image file, also called digital negative), YUV (a color encoding method), RGB (a color mode represented by three channels: red, green, and blue), and supports digital analysis.

[0084] In some embodiments, the first intermediate image corresponding to each camera is a set of distortion-free images, consistent with the number of cameras, and with the same resolution as the corresponding original image. The first intermediate images can support intuitive evaluation of the camera's intrinsic parameters and distortion parameters.

[0085] Step 102: Determine the mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera.

[0086] In some embodiments, the virtual projected image may be a BEV (Bird's Eye View, also known as a bird's-eye view) projected image.

[0087] In some embodiments, the resolution of the virtual projected image from the virtual camera is preset.

[0088] In an exemplary embodiment, the resolution of the virtual projected image of the virtual camera is set according to the hardware computing power and runtime. For example, the greater the hardware computing power and the higher the computing performance, the greater the resolution of the virtual projected image of the virtual camera; the shorter the runtime requirement, the smaller the resolution of the virtual projected image of the virtual camera.

[0089] In some embodiments, the origin Oego_rfu of the vehicle's three-dimensional coordinate system is the vertical projection ground point of the rear axle center of the vehicle, the positive direction of the Xego_rfu axis of the vehicle's three-dimensional coordinate system is directly to the right of the vehicle, the positive direction of the Yego_rfu axis of the vehicle's three-dimensional coordinate system is directly in front of the vehicle, and the positive direction of the Zego_rfu axis of the vehicle's three-dimensional coordinate system is directly above the vehicle.

[0090] In some embodiments, the virtual camera is a distortion-free camera.

[0091] In some embodiments, the origin of the image coordinate system of the virtual projected image and the origin of the image coordinate system of the first intermediate image corresponding to each camera are uniformly represented by Oimg. The two coordinate axes of the image coordinate system of the virtual projected image and the two coordinate axes of the image coordinate system of the first intermediate image are uniformly represented by the x-axis and y-axis. The origin of the pixel coordinate system of the virtual projected image and the origin of the pixel coordinate system of the first intermediate image corresponding to each camera are uniformly represented by Opixel. The two coordinate axes of the pixel coordinate system of the virtual projected image and the two coordinate axes of the pixel coordinate system of the first intermediate image are uniformly represented by the u-axis and v-axis. The positional relationship between the image coordinate system and the pixel coordinate system of the virtual projected image is consistent with the positional relationship between the image coordinate system and the pixel coordinate system of the first intermediate image. Figure 2 shows a schematic diagram of the positional relationship between the image coordinate system and the pixel coordinate system of the virtual projected image. The origin Oimg of the image coordinate system of the virtual projected image is the point (u0, v0) in the pixel coordinate system of the virtual projected image. The x-axis is parallel to the u-axis, and the y-axis is parallel to the v-axis.

[0092] In some embodiments, the origin Oimg of the image coordinate system of the virtual projection image is the projection point of the origin Oego_rfu of the vehicle's three-dimensional coordinate system in the virtual projection image, and is regarded as the reference target point.

[0093] In some embodiments, the field of view of the virtual camera is preset based on preset conditions met by the external environment. The field of view of the virtual camera is the size of the rectangular visible area projected from a top-down view by the virtual camera when it is perpendicular to the ground where the vehicle is located, representing a real 3D (three-dimensional) space.

[0094] Generally, the virtual camera's field of view is centered at the origin Oego_rfu of the vehicle's 3D coordinate system, and the aspect ratio of the virtual camera's field of view is consistent with the aspect ratio of the virtual projected image's resolution. Assuming the length of the side perpendicular to the lane lines in the virtual camera's field of view is Width_, then the length of the side parallel to the lane lines is Height_; assuming the number of pixels perpendicular to the lane lines in the virtual projected image's resolution is width_imgpixel, then the number of pixels parallel to the lane lines is height_imgpixel, and Width_ / Height_ = width_imgpixel / height_imgpixel. Generally, Height_ ranges from 30 to 150 meters. Preferably, Width_ is the actual distance between the two outermost lane lines on both sides of all lane lines in the external environment.

[0095] In some embodiments, a virtual camera capable of generating virtual projected images is constructed based on preset conditions satisfied by the external environment, the resolution of the virtual projected image, and the field of view of the virtual camera.

[0096] In some embodiments, the preset height of the virtual camera is Hvir_cam, where Hvir_cam is the vertical height from the location of the virtual camera to the ground where the vehicle is located. Generally, the minimum value of Hvir_cam is not less than the vertical height from the camera with the highest height among all cameras to the ground where the vehicle is located.

[0097] In some embodiments, the intrinsic parameters of the virtual camera are as follows:

[0098] In some embodiments, the origin Ovir of the virtual camera's 3D coordinate system is the location of the virtual camera. The Xvir axis of the virtual camera's 3D coordinate system is parallel to the x-axis of the image coordinate system of the virtual projected image. The positive direction of the Xvir axis of the virtual camera's 3D coordinate system is consistent with the positive direction of the x-axis of the image coordinate system of the virtual projected image. The Yvir axis of the virtual camera's 3D coordinate system is parallel to the y-axis of the image coordinate system of the virtual projected image. The positive direction of the Yvir axis of the virtual camera's 3D coordinate system is consistent with the positive direction of the y-axis of the image coordinate system of the virtual projected image. The Zvir axis of the virtual camera's 3D coordinate system is a straight line formed by the origin Ovir of the virtual camera's 3D coordinate system and the origin Oimg of the image coordinate system of the virtual projected image. The positive direction of the Zvir axis of the virtual camera's 3D coordinate system points from the origin Ovir of the virtual camera's 3D coordinate system to the origin Oimg of the image coordinate system of the virtual projected image.

[0099] In some embodiments, the origin Oc of the camera's 3D coordinate system represents the camera's location. The Xc axis of the camera's 3D coordinate system is parallel to the x-axis of the image coordinate system of the first intermediate image corresponding to the camera. The positive direction of the Xc axis of the camera's 3D coordinate system is consistent with the positive direction of the x-axis of the image coordinate system of the first intermediate image corresponding to the camera. The Yc axis of the camera's 3D coordinate system is parallel to the y-axis of the image coordinate system of the first intermediate image corresponding to the camera. The positive direction of the Yc axis of the camera's 3D coordinate system is consistent with the positive direction of the y-axis of the image coordinate system of the first intermediate image corresponding to the camera. The Zc axis of the camera's 3D coordinate system is a straight line formed by the origin Oc of the camera's 3D coordinate system and the origin Oimg of the image coordinate system of the first intermediate image corresponding to the camera. The positive direction of the Zc axis of the camera's 3D coordinate system points from the origin Oc of the camera's 3D coordinate system to the origin Oimg of the image coordinate system of the first intermediate image corresponding to the camera. Figure 3 shows a schematic diagram of the camera's 3D coordinate system, the image coordinate system of the first intermediate image corresponding to the camera, and the world coordinate system. In Figure 3, the origin of the world coordinate system is Ow, and the three coordinate axes of the world coordinate system are the Xw axis, Yw axis, and Zw axis. In Figure 3, the three-dimensional coordinates of the spatial three-dimensional point in the camera's three-dimensional coordinate system are P(Xc, Yc, Zc), and the two-dimensional coordinates of the imaging point of the spatial three-dimensional point in the image coordinate system of the first intermediate image corresponding to the camera are p(x, y). Point B is the intersection of the perpendicular line drawn from point P to the surface XcOcZc and the surface XcOcZc, and point A is the intersection of the perpendicular line drawn from point B to the Zc axis and the Zc axis.

[0100] In some embodiments, as shown in FIG4, step 102 includes:

[0101] Perform the following operations on any preset pixel in the virtual projected image of the virtual camera:

[0102] Step 401: Based on the first two-dimensional coordinates of the preset pixel, determine the first projection light path projected onto the preset pixel by the virtual camera.

[0103] In the exemplary embodiment, the first two-dimensional coordinates of the preset pixel are denoted as p. vir_cam (u,v), based on the first two-dimensional coordinates of the preset pixel and the intrinsic parameters of the virtual camera. vir_cam Calculate the normalized point coordinates p of the preset pixel. vir_cam (x,y), normalized point coordinates p based on preset pixel points vir_cam (x,y) determines the first projection light path that is projected onto a preset pixel point by a virtual camera.

[0104] Step 402: Determine the first three-dimensional coordinates of the target space three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera.

[0105] In some embodiments, as shown in FIG5, step 402 includes:

[0106] Step 501: Determine the first normalized vector of the first projected light path in the three-dimensional coordinate system of the virtual camera.

[0107] In an exemplary embodiment, the normalized point coordinates p of the preset pixel are used. vir_cam (x,y) determines the first normalized vector Vec of the first projected light path in the three-dimensional coordinate system of the virtual camera. vir_cam (x,y,1).

[0108] Step 502: Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

[0109] In some embodiments, as shown in FIG6, step 502 includes:

[0110] Step 601: Based on the extrinsic parameters of the virtual camera, determine the third transformation relationship from the virtual camera's 3D coordinate system to the world coordinate system.

[0111] In the exemplary embodiment, the extrinsic parameters of the virtual camera are transformed into a third transformation relationship from the virtual camera's 3D coordinate system to the world coordinate system. The third transformation relationship is a homogeneous translation and rotation matrix, and the specific formula for the third transformation relationship is as follows:

[0112] Third transformation relationship The top left 3x3 (that is, the first 3 rows and first 3 columns) represents the rotation relationship, and the third transformation relationship. The first three rows of the fourth column represent translation relationships, and the third column represents transformation relationships. The fourth line represents a homogeneous mapping.

[0113] Step 602: Determine the fourth transformation relationship from the target device's three-dimensional coordinate system to the world coordinate system.

[0114] In the exemplary embodiment, the target device is a vehicle, and the formula for the fourth transformation relationship from the vehicle's three-dimensional coordinate system to the world coordinate system is as follows:

[0115] Step 603: Based on the third and fourth transformation relationships, determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

[0116] In an exemplary embodiment, based on the third transformation relationship and the fourth transformation relationship The first transformation relationship from the vehicle's 3D coordinate system to the virtual camera's 3D coordinate system is determined. The formula for this first transformation relationship is as follows:

[0117] Based on the extrinsic parameters of the virtual camera, the third transformation relationship from the virtual camera's 3D coordinate system to the world coordinate system is determined, and the fourth transformation relationship from the target device's 3D coordinate system to the world coordinate system is determined. Based on the third and fourth transformation relationships, the first transformation relationship from the target device's 3D coordinate system to the virtual camera's 3D coordinate system is determined. With the help of the world coordinate system, the first transformation relationship from the target device's 3D coordinate system to the virtual camera's 3D coordinate system can be accurately obtained.

[0118] Step 503: Based on the first normalized vector and the first transformation relationship, determine the second normalized vector of the first projection optical path in the three-dimensional coordinate system of the target device.

[0119] In an exemplary embodiment, based on the first normalized vector Vec vir_cam (x,y,1) and the first transformation relation The second normalized vector of the first projection optical path in the three-dimensional coordinate system of the target device is determined. The formula for the second normalized vector is as follows:

[0120] in, It means The first 3 rows and first 3 columns.

[0121] Step 504: Determine the scaling factor based on the first transformation relationship and the second normalized vector.

[0122] In the exemplary embodiment, the unit vector in the positive direction of the Zego_rfu axis of the vehicle's three-dimensional coordinate system is denoted as VecGround. ego_rfu (0,0,1) is used to obtain VecGround ego_rfu(0,0,1) and the second normalized vector Vec ego_rfu The angle θ between (x′, y′, z′) is taken as... The value in the third row and fourth column is Scale. h Based on the included angle θ, Scale h Second normalized vector Vec ego_rfu (x′,y′,z′) is used to calculate the scale factor. H Scale factor H The specific calculation formula is as follows:

[0123] in ， ‖Vec ego_rfu ( x′,y′,z′)‖ Lx The second normalized vector Vec ego_rfu The norm of (x′,y′,z′) is preferably taken as the second normalized vector Vec. ego_rfu The L2 norm of (x′,y′,z′).

[0124] In the exemplary embodiment, the scale factor can be understood as the proportional relationship between the image of the virtual camera and the field of view of the virtual camera.

[0125] Step 505: Based on the first normalized vector and the scaling factor, determine the first three-dimensional coordinates of the target space three-dimensional point on the first projection light path in the three-dimensional coordinate system of the virtual camera.

[0126] In an exemplary embodiment, based on the first normalized vector Vec vir_cam (x,y,1) and scale factor Scale H Determine the first three-dimensional coordinates P of the target three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera. vir_cam (x*Scale H ,y*Scale H ,Scale H The first three-dimensional coordinates can also be represented as P. vir_cam (x*Scale H ,y*Scale H ,Scale H ,1), to maintain homogeneity with subsequent matrix calculations.

[0127] In an exemplary embodiment, a preset pixel corresponds to a fixed three-dimensional point in the target space.

[0128] The process involves determining the first normalized vector of the first projection optical path in the 3D coordinate system of the virtual camera, establishing a first transformation relationship from the 3D coordinate system of the target device to the 3D coordinate system of the virtual camera, and then determining the second normalized vector of the first projection optical path in the 3D coordinate system of the target device based on the first normalized vector and the first transformation relationship. Based on the first transformation relationship and the second normalized vector, a scaling factor is determined. Finally, based on the first normalized vector and the scaling factor, the first 3D coordinates of the target spatial 3D point on the first projection optical path in the 3D coordinate system of the virtual camera are determined. The spatial position of the first projection optical path remains unchanged; only the normalized vector of the first projection optical path differs in different 3D coordinate systems. By determining the scaling factor based on the first transformation relationship and the second normalized vector, and by using the first normalized vector and the scaling factor in the 3D coordinate system of the virtual camera, the first 3D coordinates of the target spatial 3D point on the first projection optical path in the 3D coordinate system of the virtual camera can be accurately determined.

[0129] Step 403: Based on the calibration extrinsic parameters of the target camera and the first three-dimensional coordinates, determine the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system.

[0130] The target camera is at least one camera that corresponds to a three-dimensional point in the target space.

[0131] In an exemplary embodiment, based on the first three-dimensional coordinates and the view frustum of each camera, the target camera corresponding to the three-dimensional point in the target space can be determined. Specifically, one three-dimensional point in the target space may correspond to one camera, or one three-dimensional point in the target space may correspond to two cameras, because there may be some overlap in the actual three-dimensional space captured by different cameras.

[0132] In some embodiments, as shown in FIG7, step 403 includes:

[0133] Step 701: Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

[0134] In the exemplary embodiment, the first transformation relationship is:

[0135] Step 702: Based on the calibration extrinsic parameters of the target camera, determine the second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device.

[0136] In an exemplary embodiment, a second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device is determined based on the calibration extrinsic parameters of the target camera. Where i represents the i-th camera, which is the target camera.

[0137] Step 703: Based on the first transformation relationship, the second transformation relationship, and the first three-dimensional coordinates, determine the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system.

[0138] In an exemplary embodiment, based on the first transformation relationship Second transformation relationship and the first three-dimensional coordinate P vir_cam (x*Scale H ,y*Scale H ,Scale H 1) Determine the second three-dimensional coordinates of the target space's three-dimensional points in the target camera's three-dimensional coordinate system. The specific formula for calculating the second three-dimensional coordinates is as follows:

[0139] A first transformation relationship is determined from the 3D coordinate system of the target device to the 3D coordinate system of the virtual camera. Based on the calibration extrinsic parameters of the target camera, a second transformation relationship is determined from the 3D coordinate system of the target camera to the 3D coordinate system of the target device. Based on the first transformation relationship, the second transformation relationship, and the first 3D coordinates, the second 3D coordinates of the 3D points in the target space in the 3D coordinate system of the target camera are determined. By using the 3D coordinate system of the target device, the transformation relationship between the 3D coordinate system of the target camera and the 3D coordinate system of the virtual camera can be accurately obtained. Then, based on the first 3D coordinates, the second 3D coordinates of the 3D points in the target space in the 3D coordinate system of the target camera can be accurately determined.

[0140] Step 404: Based on the second three-dimensional coordinates, determine the second projection optical path for projecting the three-dimensional points of the target space onto the target camera.

[0141] In an exemplary embodiment, based on the second three-dimensional coordinate P real_cam_i (x i ,y i ,z i The third normalized vector of the second projection ray path projected onto the target space's three-dimensional points by the target camera in the target camera's three-dimensional coordinate system is determined. The formula for the third normalized vector is as follows: Vecreal_cam_i(x i ′,y i ′,1)=Vecreal_cam_i(x i / z i ,y i / z i ,z i / z i )

[0142] In some embodiments, prior to step 404, the camera calibration evaluation method further includes: determining, based on the second three-dimensional coordinates, that a three-dimensional point in the target space can be imaged by the target camera.

[0143] In an exemplary embodiment, determining that a three-dimensional point in the target space can be imaged by a target camera based on the second three-dimensional coordinates can be achieved by using the second three-dimensional coordinates P real_cam_i (x i ,y i ,z i Zc axis coordinates z i Negative 3D points in the target space are deleted to ensure that the remaining 3D points in the target space can be imaged by the target camera.

[0144] Before determining the second projection optical path for projecting the target space 3D points onto the target space using the target camera based on the second 3D coordinates, it is determined whether the target space 3D points can be imaged by the target camera based on the second 3D coordinates. Target space 3D points that cannot be imaged by the target camera are deleted. Then, based on the second 3D coordinates, the second projection optical path for projecting the target space 3D points onto the target space using the target camera is determined. This reduces the computational amount of determining the second projection optical path and further reduces the computational amount of subsequent steps.

[0145] Step 405: Based on the second projection optical path, the intrinsic parameters of the target camera and the virtual camera, determine the second two-dimensional coordinates of the actual pixel points in the virtual projection image of the virtual camera projected onto the three-dimensional point of the target space.

[0146] In an exemplary embodiment, the third normalized vector Vercreal_cam_i(x) based on the second projection optical path in the three-dimensional coordinate system of the target camera is used. i ′,y i ′,1)Intrinsic parameters of the target camera and the virtual camera vir_cam Determine the second two-dimensional coordinates p of the actual pixel points in the virtual projection image of the virtual camera, projecting the three-dimensional points of the target space onto the target space. vir_img_i (u i ,v i ).

[0147] Step 406: Based on the second two-dimensional coordinates, determine the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera.

[0148] In an exemplary embodiment, determining the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera based on the second two-dimensional coordinates can be achieved by using the second two-dimensional coordinates p vir_img_i (u i ,v i u i and v i After decomposing these two parameters, we obtain the mapping relationship between the x-axis and y-axis directions of the first intermediate image corresponding to the target camera projected onto the virtual projection image of the virtual camera. These can be named mapx and mapy. Specifically, it can be that u...i As mapx, v i As mapy.

[0149] For any preset pixel in the virtual projection image of the virtual camera, perform the following operations: Based on the first two-dimensional coordinates of the preset pixel, determine the first projection optical path projected onto the preset pixel by the virtual camera; determine the first three-dimensional coordinates of the target space three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera; based on the calibration extrinsic parameters of the target camera and the first three-dimensional coordinates, determine the second three-dimensional coordinates of the target space three-dimensional point in the three-dimensional coordinate system of the target camera; based on the second three-dimensional coordinates, determine the second projection optical path projected onto the target space three-dimensional point by the target camera; based on the second projection optical path, the intrinsic parameters of the target camera, and the intrinsic parameters of the virtual camera, determine the second two-dimensional coordinates of the actual pixel in the virtual projection image of the virtual camera projected onto the target space three-dimensional point; based on the second two-dimensional coordinates, determine the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera, thus obtaining the mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera.

[0150] The target space 3D point lies both on the first projection optical path projected by the virtual camera and on the second projection optical path projected by the target camera. The second projection optical path can be obtained based on the first projection optical path. Then, based on the second projection optical path, the intrinsic parameters of the target camera, and the intrinsic parameters of the virtual camera, the second two-dimensional coordinates of the actual pixel points projected onto the virtual projection image of the virtual camera are determined. Based on these second two-dimensional coordinates, the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera is determined. Furthermore, the calculation process uses the calibration extrinsic parameters of the target camera; that is, the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera is related to the calibration extrinsic parameters of the target camera.

[0151] In an exemplary embodiment, as shown in Figure 8, a schematic diagram of the first and second projection optical paths is presented. In Figure 8, the three-dimensional coordinates of the target space three-dimensional point in the world coordinate system are P(Xw, Yw, Zw), pvir is a preset pixel point, Ovir is the position of the virtual camera, P(Xw, Yw, Zw), P1, P2, and P3 are points on the first projection optical path projected onto the preset pixel point pvir by the virtual camera, Oc1 is the position of the first camera, Oc1P is the second projection optical path projected by the first camera onto the target space three-dimensional point P, P(Xw, Yw, Zw) is the intersection of the first and second projection optical paths, pc1 is the imaging point of P(Xw, Yw, Zw) in the image of the first camera, and Tc1-vir is the transformation relationship from the three-dimensional coordinate system of the first camera to the three-dimensional coordinate system of the virtual camera.

[0152] Step 103: Based on the mapping relationship, the first intermediate image corresponding to each camera is mapped onto the virtual projection image to obtain the second intermediate image corresponding to each camera.

[0153] Step 104: Stitch together the second intermediate images corresponding to each camera to obtain the stitched projection image.

[0154] In an exemplary embodiment, the second intermediate images corresponding to each camera are stitched together to obtain a stitched projected image. Alternatively, the second intermediate images corresponding to each camera can be stitched together at the resolution of a virtual projected image to obtain a stitched projected image.

[0155] Step 105: Based on the stitched projected image, evaluate the calibration extrinsic parameters of each camera.

[0156] In some embodiments, step 105 includes: evaluating the calibration accuracy of the calibration extrinsic parameters of each camera based on the geometric feature relationships in the real external environment and the geometric feature relationships in the stitched projected image.

[0157] In an exemplary embodiment, evaluating the calibration accuracy of each camera's calibration extrinsic parameters can be done by evaluating whether the calibration accuracy of each camera's calibration extrinsic parameters is qualified.

[0158] In the exemplary embodiment, the stitched projected image, with the calibration accuracy of each camera's respective extrinsic parameters being qualified, satisfies the following conditions: all straight features, preferably lane lines, form continuous straight lines in the stitched projected image, and their relative positional relationships, such as parallelism or perpendicularity, are consistent with the relative positional relationships between geometric features in the real external environment; the same straight-line features in the projection areas of each camera overlap with each other without significant misalignment. Figure 9 shows a schematic diagram of a stitched projected image with the calibration accuracy of each camera's respective extrinsic parameters being qualified.

[0159] In an exemplary embodiment, as shown in Figure 10, a schematic diagram of a stitched projected image is presented where the calibration accuracy of each camera's individual calibration extrinsic parameters is substandard. Figure 10 shows a total of six cameras: a front-view camera, a left front-view camera, a right front-view camera, a left rear-view camera, a right rear-view camera, and a rear-view camera. The projected image is divided into six regions: the front-view region, the left front-view region, the right front-view region, the left rear-view region, the right rear-view region, and the rear-view region. The specific camera whose calibration extrinsic parameters are substandard can be determined based on the images of each region in the stitched projected image. In Figure 10, for the left and right rear-view areas, the parallel lane lines in the real environment are skewed in the stitched projection image and no longer maintain a parallel relationship. For the rear-view area, due to insufficient pitch angle calibration accuracy, the parallel lane lines in the real environment appear as a figure-eight shape in the stitched projection image, and the lane lines extending from the left and right sides of the vehicle tend to intersect, no longer maintaining a parallel relationship. For the front-view, left front-view, and left rear-view areas, the projections of the same lane line at the outermost edge do not coincide and intersect with each other, which indicates that the calibration accuracy is not up to standard.

[0160] In an exemplary embodiment, as shown in Figure 11, a schematic diagram of a stitched projected image is presented where the calibration accuracy of each camera's individual calibration extrinsic parameters is substandard. Figure 11 shows a total of six cameras: a front-view camera, a left front-view camera, a right front-view camera, a left rear-view camera, a right rear-view camera, and a rear-view camera. The projected image is divided into six regions: the front-view region, the left front-view region, the right front-view region, the left rear-view region, the right rear-view region, and the rear-view region. The specific camera whose calibration extrinsic parameters are substandard can be determined based on the images of each region in the stitched projected image. In Figure 11, for the right front view area, the outer lane line is skewed and no longer coincides with the same straight lane line in the front view. Similarly, for the left rear view, the outer lane line is skewed and no longer coincides with the same straight lane line in the left front and rear views. For the rear view, due to insufficient translation position calibration accuracy, the lane line shifts to the right. Due to insufficient pitch angle calibration accuracy, the parallel lane lines in the real environment appear as an inverted "V" shape in the BEV projection image, especially the outermost lane line on the right rear.

[0161] In this application, the second intermediate images corresponding to each camera are stitched together to obtain a stitched projected image. Based on the stitched projected image, the calibration extrinsic parameters of each camera are evaluated. By observing the stitched projected image from the perspective of a virtual camera, the calibration extrinsic parameters of each camera on the target device can be quickly and intuitively evaluated. This method is convenient, simple, and easy to operate, solving the problem of not being able to quickly and intuitively evaluate the calibration extrinsic parameters of multiple cameras. Moreover, it has less dependence on external environmental conditions, requiring only parallel straight line elements on a flat road surface.

[0162] In summary, this application acquires the first intermediate image corresponding to each camera on the target device, determines the mapping relationship between the first intermediate image of each camera and the virtual projection image of the virtual camera, and based on the mapping relationship, maps the first intermediate image of each camera to the virtual projection image to obtain the second intermediate image of each camera. The first intermediate images of each camera are then transformed to the virtual camera's viewpoint to obtain the second intermediate image of each camera from the virtual camera's viewpoint. The second intermediate images of each camera are then stitched together to obtain the stitched projection image. Based on the stitched projection image, the calibration extrinsic parameters of each camera are evaluated. By observing the stitched projection image from the virtual camera's viewpoint, the calibration extrinsic parameters of each camera on the target device can be quickly and intuitively evaluated. This method is convenient, simple, and easy to operate, solving the problem of the inability to quickly and intuitively evaluate the calibration extrinsic parameters of multiple cameras.

[0163] Exemplary device

[0164] Accordingly, this application also provides a camera calibration and evaluation device, as shown in FIG12, the camera calibration and evaluation device including:

[0165] The acquisition unit 1201 is configured to acquire the first intermediate image corresponding to each camera on the target device;

[0166] The processing unit 1202 is configured to determine the mapping relationship between the first intermediate image corresponding to each of the cameras and the virtual projection image of the virtual camera;

[0167] The mapping unit 1203 is configured to map the first intermediate image corresponding to each of the cameras onto the virtual projection image based on the mapping relationship, so as to obtain the second intermediate image corresponding to each of the cameras.

[0168] The stitching unit 1204 is configured to stitch together the second intermediate images corresponding to each of the cameras to obtain a stitched projection image.

[0169] Evaluation unit 1205 is configured to evaluate the calibration extrinsic parameters of each of the cameras based on the stitched projected image.

[0170] The camera calibration and evaluation apparatus provided in this embodiment belongs to the same concept as the camera calibration and evaluation method provided in the above embodiments of this application. It can execute the camera calibration and evaluation method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the camera calibration and evaluation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the camera calibration and evaluation method provided in the above embodiments of this application, and will not be repeated here.

[0171] The functions implemented by the acquisition unit 1201, processing unit 1202, mapping unit 1203, splicing unit 1204 and evaluation unit 1205 can be implemented by the same or different processors, and this application embodiment does not limit them.

[0172] It should be understood that the units in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a general-purpose central processing unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC). By designing the logical relationships between the components within the circuit, some or all of the above unit functions can be implemented. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD). Taking a field-programmable gate array (FPGA) as an example, it can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing some or all of the above unit functions. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0173] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a CPU, microprocessor, graphics processing unit (GPU), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), or deep learning processing unit (DPU).

[0174] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0175] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or for implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0176] Exemplary electronic devices

[0177] One embodiment of this application provides an electronic device, as shown in FIG13, which includes:

[0178] Memory 200 and processor 210;

[0179] The memory 200 is connected to the processor 210 and is configured to store a program;

[0180] The processor 210 is configured to implement the camera calibration and evaluation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0181] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0182] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0183] A bus can include a pathway for transmitting information between various components of a computer system.

[0184] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0185] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0186] The memory 200 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0187] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0188] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0189] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0190] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of any of the camera calibration and evaluation methods provided in the above embodiments of this application.

[0191] Exemplary computer program products and storage media

[0192] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the camera calibration and evaluation methods according to various embodiments of this application as described in any of the foregoing embodiments of this specification.

[0193] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0194] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor through steps in the camera calibration and evaluation methods according to various embodiments of this application described in any of the above embodiments of this specification. Specifically, the following steps can be implemented:

[0195] Step 101: Obtain the first intermediate image corresponding to each camera on the target device;

[0196] Step 102: Determine the mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera.

[0197] Step 103: Based on the mapping relationship, the first intermediate image corresponding to each camera is mapped to the virtual projection image to obtain the second intermediate image corresponding to each camera.

[0198] Step 104: Stitch together the second intermediate images corresponding to each camera to obtain the stitched projection image;

[0199] Step 105: Based on the stitched projected image, evaluate the calibration extrinsic parameters of each camera.

[0200] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0201] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0202] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0203] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0204] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0205] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0206] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0207] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0208] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0209] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0210] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera calibration and evaluation method, comprising: Acquire the first intermediate image corresponding to each camera on the target device; Determine the mapping relationship between the first intermediate image corresponding to each of the cameras and the virtual projection image of the virtual camera; Based on the mapping relationship, the first intermediate image corresponding to each of the cameras is mapped onto the virtual projection image to obtain the second intermediate image corresponding to each of the cameras; The second intermediate images corresponding to each of the cameras are stitched together to obtain the stitched projection image. Based on the stitched projected image, the calibration extrinsic parameters of each of the cameras are evaluated.

2. The camera calibration and evaluation method according to claim 1, wherein, Determining the mapping relationship between the first intermediate image projected onto each of the cameras and the virtual projection image of the virtual camera includes: Perform the following operations on any preset pixel in the virtual projected image of the virtual camera: Based on the first two-dimensional coordinates of the preset pixel, determine the first projection light path that is projected onto the preset pixel by the virtual camera; Determine the first three-dimensional coordinates of the target three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera; Based on the calibration extrinsic parameters of the target camera and the first three-dimensional coordinates, the second three-dimensional coordinates of the target space three-dimensional point in the three-dimensional coordinate system of the target camera are determined, wherein the target camera is at least one camera among the cameras that corresponds to the target space three-dimensional point; Based on the second three-dimensional coordinates, a second projection optical path is determined for the projection of the target space three-dimensional point onto the target camera; Based on the second projection optical path, the intrinsic parameters of the target camera, and the intrinsic parameters of the virtual camera, the second two-dimensional coordinates of the actual pixel points in the virtual projection image of the virtual camera projected onto the three-dimensional point of the target space are determined. Based on the second two-dimensional coordinates, the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera is determined.

3. The camera calibration and evaluation method according to claim 2, wherein, Determining the first three-dimensional coordinates of the target spatial three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera includes: Determine the first normalized vector of the first projection optical path in the three-dimensional coordinate system of the virtual camera; Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera; Based on the first normalized vector and the first transformation relationship, determine the second normalized vector of the first projected optical path in the three-dimensional coordinate system of the target device; Based on the first transformation relationship and the second normalized vector, the scaling factor is determined; Based on the first normalized vector and the scale factor, the first three-dimensional coordinates of the target three-dimensional point on the first projection optical path in the three-dimensional coordinate system of the virtual camera are determined.

4. The camera calibration and evaluation method according to claim 2, wherein, The step of determining the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system based on the target camera's calibration extrinsic parameters and the first three-dimensional coordinates includes: Determine the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera; Based on the calibration extrinsic parameters of the target camera, a second transformation relationship is determined from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device; Based on the first transformation relationship, the second transformation relationship, and the first three-dimensional coordinates, the second three-dimensional coordinates of the target space three-dimensional point in the target camera's three-dimensional coordinate system are determined.

5. The camera calibration and evaluation method according to claim 3 or 4, wherein, Determining the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera includes: Based on the extrinsic parameters of the virtual camera, determine the third transformation relationship from the virtual camera's three-dimensional coordinate system to the world coordinate system; Determine the fourth transformation relationship from the three-dimensional coordinate system of the target device to the world coordinate system; Based on the third and fourth transformation relationships, a first transformation relationship is determined from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

6. The camera calibration and evaluation method according to any one of claims 2 to 5, wherein, Before determining the second projection optical path for projecting the target space three-dimensional point onto the target camera based on the second three-dimensional coordinates, the method further includes: Based on the second three-dimensional coordinates, it is determined that the three-dimensional point in the target space can be imaged by the target camera.

7. The camera calibration and evaluation method according to any one of claims 1 to 6, wherein, The evaluation of the calibration extrinsic parameters of each camera based on the stitched projected image includes: The calibration accuracy of each camera's calibration extrinsic parameters is evaluated based on the geometric feature relationships in the real external environment and the geometric feature relationships in the stitched projected image.

8. A camera calibration and evaluation device, comprising: The acquisition unit is configured to acquire the first intermediate image corresponding to each camera on the target device; The processing unit is configured to determine the mapping relationship between the first intermediate image corresponding to each of the cameras and the virtual projection image of the virtual camera; The mapping unit is configured to map the first intermediate image corresponding to each of the cameras onto the virtual projection image based on the mapping relationship, so as to obtain the second intermediate image corresponding to each of the cameras. The stitching unit is configured to stitch together the second intermediate images corresponding to each of the cameras to obtain a stitched projection image. The evaluation unit is configured to evaluate the calibration extrinsic parameters of each of the cameras based on the stitched projected image.

9. An electronic device, comprising a memory and a processor; The memory is connected to the processor and is configured to store a program. The processor is configured to implement the camera calibration and evaluation method as described in any one of claims 1 to 7 by running a program in the memory.

10. A storage medium, wherein, The storage medium stores a computer program, which, when executed by a processor, implements the camera calibration and evaluation method as described in any one of claims 1 to 7.

11. A computer program product comprising computer program instructions, which, when executed by a processor, implement the camera calibration and evaluation method as described in any one of claims 1 to 7.

Images