Dynamic camera calibration method, vehicle, electronic device, and program product

Abstract

The present invention relates to the technical field of artificial intelligence, and provides a dynamic camera calibration method, a vehicle, an electronic device, and a program product. The method comprises: when a dynamic camera calibration start condition is satisfied, acquiring a reference captured image of a reference camera of a target vehicle and a target captured image of a target camera of the target vehicle; determining a lane line point set of each lane line on the basis of the reference captured image, and determining an extrinsic parameter matrix of the reference camera on the basis of the lane line point sets, each lane line point set comprising lane line pixel position coordinates of the corresponding lane line; performing feature matching on the basis of a common view area of the reference captured image and the target captured image captured at the same time to obtain a target matched feature point pair set; and determining an extrinsic parameter matrix of the target camera on the basis of the target matched feature point pair set and the extrinsic parameter matrix of the reference camera. The present invention is used to enable calibration of driving cameras of intelligent driving vehicles to achieve both efficiency and accuracy.

Description

Camera dynamic calibration methods, vehicles, electronic equipment and software products

[0001] This disclosure claims priority to Chinese patent application No. 202510039627.6, filed on January 10, 2025, entitled "Camera Dynamic Calibration Method, Vehicle, Electronic Device and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence technology, specifically to a camera dynamic calibration method, a vehicle, electronic equipment, and software products. Background Technology

[0003] Camera after-sales calibration is a crucial after-sales process for machine vision products such as autonomous vehicles and mobile robots. In the dynamic calibration solutions for autonomous vehicle cameras, some impose high requirements on road conditions and perception algorithm performance to ensure calibration accuracy, often resulting in calibration failures because road conditions cannot consistently meet these requirements, or because the required perception algorithm resources are too high for mass production deployment. Other solutions, in pursuit of efficiency, disregard the constraints of road surface elements and perception algorithms, leading to unstable calibration accuracy and impacting the safety of autonomous driving functions. Summary of the Invention

[0004] In view of the above, embodiments of this disclosure provide a camera dynamic calibration method, a vehicle, an electronic device, and a program product. The following is an overview of the subject matter described in detail in this disclosure. This overview is not intended to limit the scope of the claims.

[0005] Firstly, this disclosure provides a camera dynamic calibration method, including:

[0006] When the camera dynamic calibration start conditions are met, acquire the reference acquisition image of the reference camera of the target vehicle and the target acquisition image of the target camera.

[0007] Based on the reference acquired image, the lane line point set of each lane line is determined, and the extrinsic parameter matrix of the reference camera is determined based on the lane line point set. The lane line point set includes the lane line pixel position coordinates of the corresponding lane line.

[0008] Feature matching is performed on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a set of target matching feature point pairs;

[0009] The extrinsic matrix of the target camera is determined based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

[0010] Secondly, this disclosure provides a camera dynamic calibration device, comprising:

[0011] The acquisition module is configured to acquire the reference acquisition image of the reference camera and the target acquisition image of the target camera when the camera dynamic calibration start conditions are met.

[0012] The determination module is configured to determine the lane line point set of each lane line based on the reference acquired image, and to determine the extrinsic parameter matrix of the reference camera based on the lane line point set, wherein the lane line point set includes the lane line pixel position coordinates of the corresponding lane line.

[0013] The matching module is configured to perform feature matching based on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a set of target matching feature point pairs;

[0014] The processing module is configured to determine the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

[0015] Thirdly, this disclosure provides an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the camera dynamic calibration method as described in the first aspect.

[0019] Fourthly, this disclosure provides a computer program product, which includes a computer program that, when run in a processor, implements the camera dynamic calibration method described in the first aspect.

[0020] Optionally, the computer program may be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0021] Fifthly, this disclosure provides a vehicle configured to perform the camera dynamic calibration method described in the first aspect.

[0022] The embodiments provided in this disclosure, when the camera dynamic calibration start conditions are met, acquire a reference image from a reference camera of the target vehicle and a target image from the target camera. Based on the reference image, determine the lane line point set for each lane, and then determine the extrinsic parameter matrix of the reference camera based on the lane line point set. By determining the extrinsic parameter matrix of the reference camera through a set of image-based lane line detection methods, it does not rely on a high-performance computing platform, fully utilizes road elements, and achieves high-efficiency calibration of the reference camera. Furthermore, feature matching is performed based on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a target matching feature point pair set; based on the target matching feature point pair set and the extrinsic parameter matrix of the reference camera, the extrinsic parameter matrix of the target camera is determined. By using feature point matching and the already calibrated extrinsic parameter matrix of the reference camera, the extrinsic parameter matrix of the target camera can be efficiently determined, again without relying on a high-performance computing platform or complex processing, enabling efficient calibration of the target camera associated with the reference camera. In summary, this disclosure can achieve both calibration accuracy and efficiency on intelligent driving vehicles with low-to-medium computing power platforms, completing the calibration of a panoramic camera. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 shows a schematic flowchart of the camera dynamic calibration method in an embodiment of this disclosure.

[0025] Figure 2 shows a schematic diagram of the angle formed by the lane line and the intersection in an embodiment of this disclosure.

[0026] Figure 3 is a schematic flowchart of the calibration method for the driving camera of an intelligent driving vehicle in an embodiment of this disclosure.

[0027] Figure 4 is a block diagram of the camera dynamic calibration device in an embodiment of this disclosure.

[0028] Figure 5 shows a schematic diagram of the structure of the electronic device in an embodiment of this disclosure. Detailed Implementation

[0029] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0031] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0033] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0034] Exemplary methods

[0035] The camera dynamic calibration method provided in this disclosure can be applied to the main controller of a vehicle, or to other terminals that can communicate with the vehicle and / or sensors installed on the vehicle.

[0036] The camera dynamic calibration method provided in this embodiment, as shown in Figure 1, mainly includes the following steps:

[0037] Step 101: When the camera dynamic calibration start conditions are met, acquire the reference acquisition image of the reference camera of the target vehicle and the target acquisition image of the target camera.

[0038] In some embodiments, the reference camera includes a front-view camera, and the target camera includes a left front-view camera or a right front-view camera; and / or, the reference camera includes a rear-view camera, and the target camera includes a left rear-view camera or a right rear-view camera.

[0039] In some embodiments, satisfying the camera dynamic calibration start conditions includes satisfying the dynamic road calibration conditions, which include satisfying external road environment conditions and vehicle driving state conditions.

[0040] Meeting the external road environment conditions means that the external road environment is straight, without steep slopes, without road bumps, with multiple clear lane lines, rich features on both sides of the road, such as the presence of static objects such as trees and buildings, and good daytime lighting conditions.

[0041] Meeting the conditions for autonomous vehicle driving means that the vehicle is in human driving mode, with a speed between 30km / h and 80km / h, driving straight in the middle of the lane, without frequent left or right turns, without frequent rapid acceleration or deceleration, and without vehicle bumps. At the same time, the vehicle's two-dimensional (2D) lane line detection function can continuously, accurately, and stably output lane line point sets.

[0042] Step 102: Determine the lane line point set for each lane line based on the reference acquired image, and determine the extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes the lane line pixel position coordinates of the corresponding lane line.

[0043] In some embodiments, determining the lane line point set for each lane line based on a reference acquired image includes: detecting the lane line point set in the reference acquired image using a 2D lane line detection model, wherein the lane line point set includes the lane line pixel position coordinates.

[0044] The 2D lane detection model can be obtained by pre-training a learning model, and this disclosure does not limit the specific type of learning model.

[0045] In some embodiments, a reference acquisition image can detect multiple lane line point sets, each lane line point set corresponding to a lane line, including the pixel position coordinates of the lane line belonging to that lane line. The lane line point set of each lane line can be indexed independently.

[0046] Here, each pixel in the lane line point set is given in 2D form as the lane line pixel position coordinates, rather than 3D perceived lane lines detecting the actual physical spatial position of the lane lines. This avoids the high computational load brought by 3D perception and improves detection efficiency.

[0047] In some embodiments, determining the extrinsic parameter matrix of the reference camera based on the lane line point set includes: fitting a mathematical expression for each lane line in the image coordinate system based on the lane line point set in the reference acquired image; determining the intersection coordinates of every two lane lines in the image coordinate system based on the mathematical expression of each lane line, thus obtaining a set of intersection coordinates; determining the initial extrinsic parameter matrix of the reference camera based on the set of intersection coordinates; the initial extrinsic parameter matrix includes pitch angle, yaw angle, and default roll angle; and optimizing the initial extrinsic parameter matrix based on the mathematical expression of each lane line to obtain the optimized extrinsic parameter matrix of the reference camera.

[0048] The mathematical expression of the lane line is obtained by fitting the set of lane line points. Any fitting algorithm can be used, and there is no restriction on the specific fitting algorithm used here.

[0049] For example, after fitting a mathematical expression for each lane line in the image coordinate system based on the set of lane line points in the reference acquired image, the method further includes: removing lane lines that do not meet the conditions based on the mathematical expression of each lane line, such conditions include straight lane lines, stable driving, etc.

[0050] Remove lane lines that do not meet the conditions, including but not limited to: 1. Lane lines whose curvature parameters in their mathematical expressions exceed a certain threshold; 2. Lane lines with bifurcation points in their point sets, i.e., where one lane line becomes two lane lines, or two lane lines become one lane line; 3. Lane lines corresponding to when the vehicle is in an uphill, downhill, or continuously bumpy state, as determined by the vehicle's inertial navigation system (INS), inertial measurement unit (IMU), or combined inertial navigation information; 4. Lane lines whose length is less than a length threshold.

[0051] For example, based on the mathematical expression of each lane line, the coordinates of the intersection point of each pair of lane lines in the image coordinate system are determined, including but not limited to the following methods:

[0052] Method 1: Solve the linear regression equation of the lane line point set, and calculate the coordinates of the intersection point by using the linear regression equations of each pair of lane lines;

[0053] Method 2: Obtain two points within the preset area of ​​the lane line point set, use them as known conditions to solve for the equation of the straight line passing through these two points, and use them as the representative straight line equation of the lane line. Calculate the coordinates of the intersection point using the representative straight line equations of the two lane lines respectively.

[0054] Method 3: Solve for the equations of all straight lines from the starting point of the lane line point set to any other point in the set. Calculate the slope and offset parameters of these straight lines. Remove the maximum and minimum values ​​for both slope and offset parameters, and then calculate their averages. Substitute these average slope and offset parameter values ​​into the straight line equations to obtain the representative straight line equations for each lane line. Finally, calculate the coordinates of the intersection points using the representative straight line equations of each pair of lane lines. The point closest to the bottom of the image in each lane line point set is considered the starting point.

[0055] For example, determining the initial extrinsic parameter matrix of the reference camera based on the set of intersection point coordinates includes: determining the optimal intersection point coordinates based on the set of intersection point coordinates, and determining the initial extrinsic parameter matrix of the reference camera based on the optimal intersection point coordinates and the intrinsic parameter matrix of the reference camera.

[0056] For example, before determining the initial extrinsic matrix of the reference camera based on the optimal intersection point coordinates and the intrinsic parameter matrix of the reference camera, the method further includes: eliminating external points in the intersection point coordinate set based on known prior conditions, including the tool pose and tool accuracy of the reference camera.

[0057] In some embodiments, the method further includes: after obtaining the extrinsic parameter matrices corresponding to each of the N frame reference acquisition images, voting on the N extrinsic parameter matrices, and determining the final extrinsic parameter matrix of the reference camera based on the voting results; N is an integer greater than 1.

[0058] In some embodiments, optimizing the initial extrinsic parameter matrix based on the mathematical expression of each lane line to obtain the extrinsic parameter matrix of the reference camera includes: determining the angle between the lane line and the horizontal coordinate axis of the image coordinate system based on the mathematical expression of each lane line; and optimizing the roll angle in the initial extrinsic parameter matrix according to the angle corresponding to each lane line to obtain the extrinsic parameter matrix of the reference camera.

[0059] As shown in Figure 2, the angle between lane line LineLeft_i and the horizontal coordinate axis of the image coordinate system (i.e., the horizontal dashed line) is θ1, the angle between lane line LineRight_i and the horizontal coordinate axis of the image coordinate system (i.e., the horizontal dashed line) is θ2, the intersection point of LineLeft_i and LineRight_i is VP, and the angle formed by LineLeft_i and LineRight_i at the intersection point is θ3.

[0060] Ideally, θ1 and θ2 are equal. Here, the roll angle of the reference camera is optimized based on the difference between θ1 and θ2. The larger the difference between θ1 and θ2, the larger the optimized roll angle.

[0061] For example, the optimal angle of a lane line is determined based on the angles of lane lines that are statistically identified, and the roll angle in the initial extrinsic parameter matrix is ​​optimized based on the optimal angle of each lane line.

[0062] Step 103: Perform feature matching based on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a set of target matching feature point pairs.

[0063] In some embodiments, feature matching is performed based on the common viewing region of a reference image and a target image acquired at the same time to obtain a target matching feature point pair set, including: cropping the reference image and the target image to obtain a first intermediate image group; the first intermediate image group includes a reference intermediate image obtained by cropping the reference image and a target intermediate image obtained by cropping the target image, both the reference intermediate image and the target intermediate image include the common viewing region of the reference image and the target image; for the first intermediate image group, feature points of the reference intermediate image and the target intermediate image are detected, and feature matching is performed based on the feature points of the reference intermediate image and the target intermediate image to obtain a target matching feature point pair set.

[0064] To ensure consistency, computational efficiency, and detection stability, the images in the first intermediate image group are kept to be of the same size and their ranges include the common viewing areas of the associated cameras. In other words, the common viewing areas of the two associated cameras are a subset of the cropped image ranges.

[0065] In some embodiments, feature matching is performed based on feature points of a reference intermediate image and a target intermediate image to obtain a target matching feature point pair set, including: determining the matching degree between feature point pairs of the reference intermediate image and the target intermediate image, obtaining feature point pairs with matching degrees higher than a threshold, and obtaining a first matching feature point group; determining the norm of feature point pairs in the first matching feature point group, deleting feature point pairs with norms exceeding the norm threshold from the first matching feature point group, and obtaining a second matching feature point group; using feature point pairs in the second matching feature point group as target matching feature point pairs, and obtaining a target matching feature point pair set.

[0066] By acquiring feature point pairs with a matching degree higher than a threshold, a first matching feature point group is formed. This ensures that the first matching feature point group consists of the remaining feature point pairs after removing false detections and false matches, thus guaranteeing the accuracy of the calculation.

[0067] For example, the norm is the L2 norm in the pixel coordinate system, and the norm threshold is pre-configured based on the field of view of the images in the first intermediate image group and the debugging statistics.

[0068] Step 104: Determine the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

[0069] In some embodiments, determining the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera includes: determining a base matrix based on the target matching feature point pair set when the number of target matching feature point pairs in the target matching feature point pair set is not less than a preset number; determining feature point pairs in the target matching feature point pair set that do not satisfy epipolar geometric projection based on the base matrix, and deleting them from the target matching feature point pair set to obtain a third matching feature point group; determining the relative extrinsic matrix of the target camera pose relative to the origin of the reference camera based on the third matching feature point group; and determining the extrinsic matrix of the target camera based on the extrinsic matrix of the reference camera and the relative extrinsic matrix of the target camera.

[0070] The preset quantity is pre-configured. For example, the preset quantity is 7 or 8, that is, the number of target matching feature point pairs in the target matching feature point pair set is not less than 7 or 8 feature point pairs.

[0071] By filtering out feature point pairs that do not satisfy epipolar geometric projection, and then further eliminating feature pairs that do not satisfy projection relations, the feature point pairs in the third matching feature point group can satisfy projection relations, thus further ensuring the accuracy of extrinsic parameter calibration.

[0072] For example, determining the relative extrinsic parameter matrix of the target camera relative to the origin pose of the reference camera based on the third set of matched feature points includes: solving the relative extrinsic parameter matrix of the target camera based on the origin pose of the reference camera using a nonlinear optimization algorithm based on the third set of matched feature points.

[0073] For example, the Ceres solver library can be used to solve nonlinear optimization algorithms. Specifically, one optimization model in the Ceres solver library is the Bundle Adjustment (BA) method, which aims to minimize reprojection error. Given the mounting poses of the target and reference cameras and the coordinates of each feature point in the third matching feature point set, these feature points are projected from the reference camera image plane to the target camera image plane using projection equations. The reprojection error between these predicted points and the actual observed points can then be calculated. Optimization algorithms (such as Gauss-Newton, Levenberg-Marquardt, and trust region methods) are used to iteratively adjust the extrinsic parameter matrix of the target camera to reduce the reprojection error. In each iteration, the gradient is calculated based on the current error, and the extrinsic parameter matrix of the target camera is updated in the opposite direction of the gradient. This process is repeated until the reprojection error reaches an acceptable level or a preset number of iterations is reached. Finally, an optimized set of camera extrinsic parameter matrices is obtained through iterative calculation. This set of parameters more accurately describes the extrinsic parameter matrix of the target camera relative to the origin pose of the reference camera.

[0074] For example, determining the extrinsic matrix of the target camera based on the extrinsic matrix of the reference camera and the relative extrinsic matrix of the target camera includes: using the extrinsic matrix of the reference camera as a reference, transforming the relative extrinsic matrix of the target camera to the vehicle coordinate system to maintain consistency with the reference camera, thereby obtaining the extrinsic matrix corresponding to the image acquired in the current frame of the target camera.

[0075] For example, assuming the reference camera is a forward-looking camera, This represents the transformation matrix from the forward-looking camera coordinate system to the vehicle's "Right-Forward-Up" (RFU) coordinate system, starting from a point P in the forward-looking camera coordinate system. front_cam The coordinates of this point in the vehicle's RFU coordinate system can be obtained. Assuming the target camera is any one-view camera (left or right front-view camera) that shares a field of view with the front-view camera, This represents the transformation matrix from the panoramic camera coordinate system to the forward-looking camera coordinate system, starting from a point P in the panoramic camera coordinate system. side_cam The coordinates of this point in the forward-looking camera coordinate system can be obtained. So, This represents the transformation matrix from the panoramic camera coordinate system to the vehicle RFU coordinate system. A point P in the panoramic camera coordinate system side_cam The coordinates P of this point in the vehicle's RFU coordinate system can be obtained. rfu = When the reference camera is a rear-view camera, the transformation matrix between the rear-view camera and the panoramic camera, as well as the vehicle's RFU coordinate system, is similar.

[0076] In some embodiments, the method further includes: when obtaining M extrinsic parameter matrices of the target camera by means of reference acquisition images and target acquisition images acquired at the same time in consecutive M frames, voting on the M extrinsic parameter matrices to determine the final extrinsic parameter matrix of the target camera; M is an integer greater than 1.

[0077] For example, before the voting decision, outliers in the obtained extrinsic parameter matrix are removed to eliminate abnormal and out-of-limit data. Here, abnormal and out-of-limit data refers to any component of the pitch angle, yaw angle, and roll angle of the extrinsic parameter matrix exceeding a preset range, in which case the extrinsic parameter matrix is ​​identified as an outlier.

[0078] In some embodiments, after obtaining the extrinsic parameter matrix of the target camera, the extrinsic parameter matrix of the target camera is compared with the tool pose. If the difference between the two is within the allowable range, epipolar geometry verification is performed based on the extrinsic parameter matrix. If the verification passes, the extrinsic parameter matrix is ​​saved and logged. If the result of the comparison with the tool pose is not within the allowable range, the extrinsic parameter matrix and the corresponding error code are saved in the calibration process record. If the epipolar geometry verification fails, the extrinsic parameter matrix and the corresponding error code are also saved in the calibration process record.

[0079] In an exemplary embodiment, as shown in Figure 3, the calibration method for a driving camera in an intelligent driving vehicle mainly includes:

[0080] Step 301: Obtain the lane line point set of each frame in the image sequence acquired by the front-view camera and the rear-view camera;

[0081] Step 302: Calculate the extrinsic matrix of the front-view camera based on the lane line point set of the front-view camera, and calculate the extrinsic matrix of the rear-view camera based on the lane line point set of the rear-view camera.

[0082] Step 303: For the left and right front-view cameras that share a viewing area with the front-view camera, acquire the first shared viewing area image of the left front-view camera and the front-view camera, and the second shared viewing area image of the right front-view camera and the front-view camera, respectively; and for the left and right rear-view cameras that share a viewing area with the rear-view camera, acquire the third shared viewing area image of the left rear-view camera and the rear-view camera, and the fourth shared viewing area image of the right rear-view camera and the rear-view camera, respectively.

[0083] Step 304: By cropping, obtain the first intermediate image group corresponding to the first shared viewing area image, the first intermediate image group corresponding to the second shared viewing area image, the first intermediate image group corresponding to the third shared viewing area image, and the first intermediate image group corresponding to the fourth shared viewing area image.

[0084] Step 305: Solve the extrinsic parameter matrix of the left front-view camera based on the first intermediate image group corresponding to the first common view area image; solve the extrinsic parameter matrix of the right front-view camera based on the first intermediate image group corresponding to the second common view area image; solve the extrinsic parameter matrix of the left rear-view camera based on the first intermediate image group corresponding to the third common view area image; and solve the extrinsic parameter matrix of the right rear-view camera based on the first intermediate image group corresponding to the fourth common view area image.

[0085] Step 306: Process the calibration results of each vehicle camera to complete the calibration process.

[0086] The embodiments provided in this disclosure, when the camera dynamic calibration start conditions are met, acquire a reference image from a reference camera of the target vehicle and a target image from the target camera. Based on the reference image, determine the lane line point set for each lane, and then determine the extrinsic parameter matrix of the reference camera based on the lane line point set. By determining the extrinsic parameter matrix of the reference camera through a set of image-based lane line detection methods, it does not rely on a high-performance computing platform, fully utilizes road elements, and achieves high-efficiency calibration of the reference camera. Furthermore, feature matching is performed based on the common viewing area of ​​the reference and target images acquired at the same time to obtain a set of target matching feature point pairs; the extrinsic parameter matrix of the target camera is determined based on the target matching feature point pair set and the extrinsic parameter matrix of the reference camera. By using feature point matching and the already calibrated extrinsic parameter matrix of the reference camera, the extrinsic parameter matrix of the target camera can be efficiently determined, again without relying on a high-performance computing platform or complex processing, enabling efficient calibration of the target camera associated with the reference camera. In summary, this disclosure can achieve both calibration accuracy and efficiency for panoramic cameras on intelligent driving vehicles with low to medium computing power platforms.

[0087] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between steps is not limited to implementation according to step number.

[0088] Exemplary device

[0089] Figure 4 is a block diagram of a camera dynamic calibration device provided in an embodiment of the present disclosure. The camera dynamic calibration device mainly includes: an acquisition module 401, which is configured to acquire a reference acquisition image of a reference camera of a target vehicle and a target acquisition image of a target camera when the camera dynamic calibration start conditions are met.

[0090] The determination module 402 is configured to determine the lane line point set of each lane line based on the reference acquired image, and determine the extrinsic parameter matrix of the reference camera based on the lane line point set; the lane line point set includes the lane line pixel position coordinates of the corresponding lane line.

[0091] The matching module 403 is configured to perform feature matching based on the common viewing area of ​​the reference acquisition image and the target acquisition image acquired at the same time to obtain a set of target matching feature point pairs;

[0092] Processing module 404 is configured to determine the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

[0093] Exemplary electronic devices

[0094] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0095] Referring to FIG5, an embodiment of this disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by at least one processor 501, and the one or more computer programs are executed by at least one processor 501 to enable at least one processor 501 to perform the above-described camera dynamic calibration method.

[0096] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0097] Exemplary vehicles, computer program products, and storage media

[0098] This disclosure also provides a vehicle configured to perform the camera dynamic calibration method as described above.

[0099] This disclosure also provides a computer program product, including a computer program that, when run in a processor, implements the above-described camera dynamic calibration method.

[0100] Computer programs can be stored on readable storage media of a computer device or in the cloud; the processor of a computer device reads computer programs from readable storage media or in the cloud.

[0101] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically manifested as a computer storage medium; in another optional embodiment, the computer program product is specifically manifested as a software product, such as a software development kit (SDK), etc.

[0102] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0103] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0105] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0106] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0107] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0111] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A camera dynamic calibration method, comprising: When the camera dynamic calibration start conditions are met, acquire the reference acquisition image of the reference camera of the target vehicle and the target acquisition image of the target camera. Based on the reference acquired image, the lane line point set of each lane line is determined, and the extrinsic parameter matrix of the reference camera is determined based on the lane line point set. The lane line point set includes the lane line pixel position coordinates of the corresponding lane line. Feature matching is performed on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a set of target matching feature point pairs; The extrinsic matrix of the target camera is determined based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

2. The method according to claim 1, wherein, Determining the extrinsic parameter matrix of the reference camera based on the lane line point set includes: Based on the set of lane line points in the benchmark image, a mathematical expression for each lane line in the image coordinate system is fitted. Based on the mathematical expression for each lane line, the coordinates of the intersection point of each pair of lane lines in the image coordinate system are determined, resulting in a set of intersection point coordinates. The initial extrinsic parameter matrix of the reference camera is determined based on the set of intersection point coordinates; the initial extrinsic parameter matrix includes pitch angle, yaw angle and default roll angle; The initial extrinsic parameter matrix is ​​optimized based on the mathematical expression for each lane line to obtain the optimized extrinsic parameter matrix of the reference camera.

3. The method according to claim 2, wherein, The method further includes: After obtaining the extrinsic parameter matrices corresponding to each of the N frames of the reference acquisition images, a vote is taken on the N extrinsic parameter matrices, and the final extrinsic parameter matrix of the reference camera is determined based on the voting results; where N is an integer greater than 1.

4. The method according to claim 2 or 3, wherein, The optimization of the initial extrinsic parameter matrix based on the mathematical expression for each lane line yields the optimized extrinsic parameter matrix of the reference camera, including: Based on the mathematical expression for each lane line, the angle between the lane line and the horizontal coordinate axis of the image coordinate system is determined. Based on the included angle corresponding to each lane line, the roll angle in the initial extrinsic parameter matrix is ​​optimized to obtain the extrinsic parameter matrix of the reference camera.

5. The method according to any one of claims 1 to 4, wherein, The feature matching based on the shared viewing region of the reference image and the target image acquired at the same time is used to obtain a set of target matching feature point pairs, including: The reference acquisition image and the target acquisition image are cropped to obtain a first intermediate image group. The first intermediate image group includes a reference intermediate image obtained by cropping the reference acquisition image and a target intermediate image obtained by cropping the target acquisition image. Both the reference intermediate image and the target intermediate image include the common viewing area of ​​the reference acquisition image and the target acquisition image. For the first intermediate image group, feature points of the reference intermediate image and the target intermediate image are detected, and feature matching is performed based on the feature points of the reference intermediate image and the target intermediate image to obtain a target matching feature point pair set.

6. The method according to claim 5, wherein, The step of performing feature matching based on feature points of the reference intermediate image and the target intermediate image to obtain a target matching feature point pair set includes: Determine the matching degree between feature point pairs of the reference intermediate image and the target intermediate image, obtain feature point pairs with matching degree higher than a threshold, and obtain the first matching feature point group; Determine the norm of the feature point pairs in the first matching feature point group, and delete the feature point pairs whose norm exceeds the norm threshold from the first matching feature point group to obtain the second matching feature point group. The feature point pairs in the second matching feature point group are used as target matching feature point pairs to obtain the target matching feature point pair set.

7. The method according to any one of claims 1 to 6, wherein, The step of determining the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera includes: When the number of target matching feature point pairs in the target matching feature point pair set is not less than a preset number, the basic matrix is ​​determined according to the target matching feature point pair set. Based on the fundamental matrix, identify feature point pairs in the target matching feature point pair set that do not satisfy epipolar geometric projection, and delete them from the target matching feature point pair set to obtain the third matching feature group; Based on the third set of matching feature points, the relative extrinsic parameter matrix of the target camera pose relative to the origin of the reference camera is determined; The extrinsic parameter matrix of the target camera is determined based on the extrinsic parameter matrix of the reference camera and the relative extrinsic parameter matrix of the target camera.

8. The method according to claim 7, wherein, The method further includes: When obtaining M extrinsic parameter matrices of the target camera by acquiring the reference acquisition image and the target acquisition image at the same time in consecutive M frames, the final extrinsic parameter matrix of the target camera is determined by voting on the M extrinsic parameter matrices; where M is an integer greater than 1.

9. The method according to any one of claims 1 to 8, wherein, The reference camera includes a forward-looking camera, and the target camera includes a left forward-looking camera or a right forward-looking camera; and / or The reference camera includes a rear-view camera, and the target camera includes a left rear-view camera or a right rear-view camera.

10. A camera dynamic calibration device, comprising: The acquisition module is configured to acquire the reference acquisition image of the reference camera and the target acquisition image of the target camera when the camera dynamic calibration start conditions are met. The determination module is configured to determine the lane line point set of each lane line based on the reference acquired image, and to determine the extrinsic parameter matrix of the reference camera based on the lane line point set, wherein the lane line point set includes the lane line pixel position coordinates of the corresponding lane line. The matching module is configured to perform feature matching based on the common viewing area of ​​the reference image and the target image acquired at the same time to obtain a set of target matching feature point pairs; The processing module is configured to determine the extrinsic matrix of the target camera based on the target matching feature point pair set and the extrinsic matrix of the reference camera.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the camera dynamic calibration method as described in any one of claims 1-9.

12. A computer program product, wherein, The computer program product includes a computer program that, when run in a processor, implements the camera dynamic calibration method according to any one of claims 1 to 9.

13. A vehicle, wherein, The vehicle is configured to perform the camera dynamic calibration method according to any one of claims 1 to 9.

Images