Image distortion correction method and apparatus, electronic device, and storage medium

Abstract

An image distortion correction method and apparatus, an electronic device, and a storage medium. The image distortion correction method comprises: obtaining a first image of a calibration plate photographed by a camera; for each preset feature point, determining a current feature point in the first image corresponding to the preset feature point, the preset feature point being a feature point obtained by mapping the feature point on the calibration plate to the image photographed by the camera on the basis of a positional relationship between the camera and the calibration plate when the camera photographs the first image; on the basis of the preset feature point and a predicted feature point, determining a value of a distortion parameter of the camera, the predicted feature point being indicated by a current feature point in the first image and an undetermined distortion parameter of the camera; and correcting the first image by using the distortion parameter of the determined value of the camera, so as to obtain a second image. According to the method, distortion correction can be accurately performed on the image photographed by the camera.

Description

Image distortion correction methods, devices, electronic equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411832401.3, filed on December 12, 2024, entitled “Image Distortion Correction Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision technology, specifically to an image distortion correction method, an image distortion correction device, an electronic device, a storage medium, and a computer program product. Background Technology

[0003] Lens distortion refers to the geometric deformation of image information caused by the optical characteristics of the lens used to capture images. It mainly includes radial distortion and tangential distortion. Radial distortion usually manifests as barrel distortion (image information at the image edges bulges outward) and pincushion distortion (image information at the image edges concave inward), while tangential distortion is an asymmetric distortion caused by the lens and the imaging plane not being perfectly parallel.

[0004] Lens distortion causes straight lines in an image to become curved, resulting in image distortion that affects the geometric accuracy of the image and subsequent computer vision tasks. To correct these distortions, lens distortion calibration is usually required to adjust the accuracy using the camera's internal and external parameters. Calibration involves taking multiple images of a stereo calibration block or a calibration board in various orientations to estimate distortion parameters.

[0005] When the camera's depth of field is shallow, tilting the calibration plate can cause out-of-focus issues. This makes it difficult to calibrate the camera's internal and external parameters, resulting in poor distortion calibration accuracy and affecting image distortion correction. Summary of the Invention

[0006] This application is made in consideration of the above-mentioned issues.

[0007] According to a first aspect of this application, an image distortion correction method is provided. The method includes: acquiring a first image of a calibration board captured by a camera; for each current feature point in the first image, determining a preset feature point corresponding to the current feature point, wherein the preset feature point is based on the positional relationship between the camera and the calibration board when the first image is captured, mapping feature points on the calibration board to feature points in the first image; determining the value of a distortion parameter of the camera based on the preset feature point and a predicted feature point, wherein the predicted feature point is represented using the current feature point in the first image and the undetermined distortion parameter of the camera; and correcting the first image using the determined distortion parameter of the camera to obtain a second image.

[0008] In one possible implementation, determining the value of the camera's distortion parameter based on the preset feature points and predicted feature points includes: determining a minimization function for the unknown quantity of the distortion parameter based on the variance of the coordinates between the predicted feature points and the preset feature points corresponding to the predicted feature points; performing at least one line search on the minimization function to minimize the function; and when the minimization function reaches the minimum value, using the current value of the distortion parameter as the value of the distortion parameter.

[0009] In one possible implementation, performing at least one line search on the minimization function to minimize the function includes: determining the search direction for each line search using a quasi-Newton method; performing at least one line search on the minimization function according to the determined search direction, and updating the value of the distortion parameter after each line search until the minimization function reaches the minimum value.

[0010] In one possible implementation, correcting the first image using a predetermined distortion parameter of the camera to obtain the second image includes: determining corrected pixels in the first image based on the predetermined distortion parameter; and determining the second image based on the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of the pixels adjacent to the corrected pixel.

[0011] In one possible implementation, the first image is taken using the camera with the camera's optical axis perpendicular to the calibration plate.

[0012] In one possible implementation, the camera's flange interface is provided with a reflector, and the method further includes: illuminating the reflector with a laser autocollimator and receiving the reflected light signal reflected from the reflector, wherein the laser autocollimator is configured such that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located; and determining whether the optical axis of the camera is perpendicular to the calibration plate based on the reflected light signal received by the laser autocollimator.

[0013] In one possible implementation, the calibration board is configured such that the grid pattern of the calibration board in the first image is arranged parallel to the horizontal direction of the plane in which the first image is located.

[0014] In one possible implementation, the method further includes: determining the standard deviation between the side length of each checkerboard grid in the second image and the average value of the side length of each checkerboard grid; and when the standard deviation is less than a preset threshold, correcting other images taken with the camera in the same pose using the determined distortion parameter.

[0015] In one possible implementation, the unknown distortion parameters include the camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

[0016] According to a second aspect of this application, an image distortion correction apparatus is also provided, comprising:

[0017] The image receiving module is used to acquire the first image of the calibration board captured by the camera;

[0018] The feature point matching module is used to determine the current feature point in the first image corresponding to each preset feature point. The preset feature point is based on the positional relationship between the camera and the calibration board when the camera captures the first image, and maps the feature points on the calibration board to the feature points in the image captured by the camera.

[0019] The parameter determination module is used to determine the value of the distortion parameter of the camera based on the preset feature points and the predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the undetermined distortion parameters of the camera.

[0020] An image correction module is used to correct the first image using the camera's pre-determined distortion parameters to obtain a second image.

[0021] According to a third aspect of this application, an electronic device is also provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the image distortion correction method described above.

[0022] According to a fourth aspect of this application, a storage medium is also provided, on which program instructions are stored, which, when executed, are used to perform the image distortion correction method described above.

[0023] According to a fifth aspect of this application, a computer program product is also provided, including computer program instructions that, when executed, perform the image distortion correction method described above.

[0024] In the above technical solution, for each current feature point in the first image, a preset feature point corresponding to that current feature point is determined. The preset feature point is based on the positional relationship between the camera and the calibration board when the first image is captured, mapping feature points on the calibration board to feature points in the first image. Then, based on the current feature point in the first image, the predicted feature point using the camera's undetermined distortion parameters, and the preset feature point, the value of the camera's distortion parameters is determined. Finally, using the determined distortion parameters of the camera, the first image is corrected to obtain the second image. This allows for accurate distortion correction of images captured by the camera.

[0025] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0026] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 shows a schematic flowchart of an image distortion correction method according to an embodiment of this application;

[0028] Figure 2 shows a schematic flowchart of determining the values ​​of camera distortion parameters based on preset feature points and predicted feature points according to an embodiment of this application.

[0029] Figure 3 shows a schematic flowchart of performing at least one line search on a minimization function according to an embodiment of the present application so that the minimization function obtains a minimum value;

[0030] Figure 4 shows a schematic flowchart of performing at least one line search on a minimization function according to another embodiment of the present application so that the minimization function obtains its minimum value;

[0031] Figure 5 shows a schematic flowchart of correcting a first image to obtain a second image using a determined value of distortion parameters of a camera according to an embodiment of this application.

[0032] Figure 6 shows a schematic flowchart of whether the optical axis of the camera is perpendicular to the calibration plate according to an embodiment of the present application;

[0033] Figure 7 shows a schematic diagram of determining whether the optical axis of a camera is perpendicular to the calibration plate using a laser autocollimator according to an embodiment of this application;

[0034] Figure 8 shows a schematic diagram of a first image according to an embodiment of this application;

[0035] Figure 9 shows a schematic flowchart of an image corrected using required distortion parameters according to an embodiment of this application;

[0036] Figure 10 shows a schematic flowchart of an image distortion correction method according to yet another embodiment of this application;

[0037] Figure 11 shows a schematic block diagram of an image distortion correction apparatus according to an embodiment of this application; and

[0038] Figure 12 shows a schematic block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0040] To at least partially address the aforementioned problems, a first aspect of this application proposes an image distortion correction method. This method, for each current feature point in a first image, determines a preset feature point corresponding to that current feature point. The preset feature point is based on the positional relationship between the camera and a calibration board when the first image is captured, mapping feature points on the calibration board to feature points in the first image. Then, based on the current feature point in the first image, a predicted feature point using undetermined distortion parameters of the camera, and the preset feature point, the value of the camera's distortion parameters is determined. Finally, the first image is corrected using the determined distortion parameters of the camera to obtain a second image.

[0041] Figure 1 shows a schematic flowchart of an image distortion correction method according to an embodiment of this application. As shown in Figure 1, the image distortion correction method may include steps S110 to S140.

[0042] In step S110, the first image of the calibration board captured by the camera is obtained.

[0043] The camera can be a shallow depth-of-field camera, such as a microscope. It can be used to photograph a designated area of ​​the calibration board to obtain a first image. This designated area can be the entire calibration board or a portion thereof. When taking the first image of the calibration board, the position between the camera and the calibration board should be fixed. This reduces the steps required for calibrating the camera's intrinsic and extrinsic parameters, thus minimizing the impact of these calibration steps on the distortion correction accuracy.

[0044] When capturing the first image using this camera, the angle between the plane of the calibration plate and the camera's optical axis should be as close to perpendicular as possible. This minimizes the impact of camera intrinsic and extrinsic parameter calibration on distortion correction accuracy. The image information in the first image should be clear and accurate.

[0045] In one possible implementation, the first image is taken using the camera with the camera's optical axis perpendicular to the calibration plate.

[0046] When the camera's optical axis is perpendicular to the calibration plate, distortion parameters are determined with a relatively ideal camera orientation, eliminating the need to calibrate the camera's intrinsic and extrinsic parameters. This reduces errors caused by calibrating these parameters and results in higher accuracy of the distortion parameters.

[0047] In step S120, for each current feature point in the first image, a preset feature point corresponding to the current feature point is determined. The preset feature point is based on the positional relationship between the camera and the calibration board when the camera captures the first image, and the feature points on the calibration board are mapped to the feature points in the first image.

[0048] The current feature point in the first image is a point that represents the image information of the first image, such as corner points, key points, and feature points extracted from the feature extraction. Feature points can also be manually specified points. After correcting the image distortion of the first image, the size of the first image does not change; therefore, the pixel coordinate range of the first and second images is the same in the image coordinate system corresponding to the camera used to capture the first image. For example, the current feature point in the first image can be a corner point in the first image. For a calibration board in the first image, the corner points in the first image are typically the corner points of the checkerboard grid on that calibration board.

[0049] Based on the positional relationship between the camera and the calibration board when the first image is captured, an image coordinate system is established using the plane containing the first image and the plane containing the calibration board, thereby establishing a camera imaging model. According to a specified area of ​​the calibration board in the first image, the corresponding calibration board region is determined in the world coordinate system of the calibration board. For each feature point in this calibration board region, based on the mapping relationship between the calibration board coordinate system and the image coordinate system of the first image when the camera captures the first image, a preset feature point in the first image is mapped to each feature point in the calibration board region, and the pixel coordinates of these preset feature points are determined. Here, the preset feature points are not real feature points in the first image, but are only used as virtual feature points corresponding to the current feature point in the first image. It can be understood that the current feature point in the first image corresponds to the feature point in the calibration board region corresponding to the specified area of ​​the calibration board in the first image, and the feature point in the calibration board region corresponding to the specified area of ​​the calibration board in the first image corresponds to the preset feature point. Therefore, when both the current feature point and the preset feature point correspond to the same feature point in the calibration board region corresponding to the specified area of ​​the calibration board in the first image, the current feature point and the preset feature point are also corresponding. After determining the current feature point and the preset feature point, the user can specify the current feature point and its corresponding preset feature point.

[0050] When the camera capturing the first image has no lens distortion, there is also no image distortion in the first image. In this case, the pixel coordinates of the current feature point and its corresponding preset feature point in the first image are the same. However, because the first image contains image distortion, this distortion affects the pixel coordinates of the current feature point, causing them to differ from the preset feature point. However, the impact of image distortion on the pixel coordinates of the current feature point may be minor. For example, based on the distance between the pixel coordinates of the current feature point and its corresponding preset feature point in the first image, the preset feature point with the smallest distance to the current feature point can be determined as the preset feature point corresponding to the current feature point.

[0051] In step S130, the values ​​of the camera's distortion parameters are determined based on preset feature points and predicted feature points, wherein the predicted feature points are represented using the current feature points in the first image and the undetermined distortion parameters of the camera.

[0052] The difference between the current feature point and the predicted feature point characterizes the impact of the distortion parameter on the image information of the first image, i.e., the degree of image distortion. The predicted feature point is the point corresponding to the current feature point after image distortion correction. Because the value of the distortion parameter is undetermined, the position of the predicted feature point corresponding to the current feature point is also undetermined at this time. The target position of the predicted feature point can be determined based on the position to which the current feature point needs to be corrected (e.g., the position of the preset feature point corresponding to the current feature point in the first image). Then, the value of the unknown distortion parameter can be deduced from the target position and the position of the current feature point.

[0053] In one possible implementation, the distortion parameters include the camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

[0054] First, the pixel coordinates of the current feature point can be converted into corresponding normalized camera coordinates. Then, based on the normalized camera coordinates of the current feature point and the camera's distortion parameters, the normalized camera coordinates of the predicted feature points corresponding to each current feature point can be determined. Finally, the pixel coordinates of the predicted feature points can be determined based on their normalized coordinates.

[0055] For example, the normalized camera coordinates (x, y) of the current feature point can be determined according to the following formulas 1 and 2. distorted y distorted ): x distorted =(uc x ) / f x Formula 1 y distorted =(vc y ) / f y Formula 2

[0056] Where u represents the horizontal pixel coordinate of the current feature point, v represents the vertical pixel coordinate of the current feature point, and c x c y c represents the pixel coordinates of the camera principal point (the intersection of the optical axis and the image plane corresponding to the first image). x The coordinates of the principal point in the horizontal direction are c. y These are the coordinates of the principal point in the vertical direction, usually located at the center of the image. x f y This indicates the camera's focal length, measured in pixels (f). x f represents the focal length of the camera in the horizontal direction. y This indicates the focal length in the vertical direction of the camera.

[0057] The camera that captures the first image may have radial and tangential distortion. Therefore, the normalized camera coordinates corresponding to the predicted feature points of the current feature points can be determined according to Formula 1 above, the radial distortion coefficient, and the tangential distortion coefficient.

[0058] For example, in determining the normalized camera coordinates (x, y) of the current feature point distorted y distorted After that, the normalized camera coordinates (x, y) corresponding to the predicted feature points of the current feature points can be determined according to the following formulas 3, 4 and 5. corrected y corrected ): x corrected =[x distorted -2p1x distorted y distorted -p2(r 2 +2x distorted 2 )] / (1+k1r 2 +k2r 4 +k3r 6 ) Formula 4 y corrected =[y distorted -2p1x distorted y distorted -p2(r 2 +2y distorted 2 )] / (1+k1r 2 +k2r 4 +k3r 6 ) Formula 5

[0059] Where k1, k2, and k3 represent radial distortion parameters, p1 and p2 represent tangential distortion parameters, and r represents the radial distance from the normalized camera coordinates of the current feature point to the camera optical axis.

[0060] After determining the normalized camera coordinates (x) corresponding to the predicted feature points. distorted y distorted After that, it needs to be converted into the corresponding pixel coordinates (u). corrected v corrected ).

[0061] For example, the pixel coordinates corresponding to the predicted feature points can be determined according to the following formulas 6 and 7: u corrected =x corrected *f x +c x Formula 6 v corrected =y corrected *f y +c y Formula 7

[0062] For each predicted feature point, the position of the preset feature point corresponding to that predicted feature point in the first image can be used as the target position where the predicted feature point should be in the second image. This target position can be used as the position of the predicted feature point. Based on the positional difference between the predicted feature point and the current feature point corresponding to it, the value of the distortion parameter corresponding to the current feature point in the first image can be determined.

[0063] However, there are multiple current feature points in the first image. If each current feature point is corrected to the position of its corresponding preset feature point, the determined distortion parameter values ​​may not be the same. Therefore, in order to correct the image distortion in the first image, the optimal distortion parameter values ​​can be determined based on all the multiple current feature points, so as to make the predicted feature point corresponding to each current feature point as close as possible to its corresponding preset feature point. This can ensure that the correction effect on the image distortion in the first image is optimal.

[0064] In step S140, the first image is corrected using the camera's known distortion parameters to obtain the second image.

[0065] The determined distortion parameter characterizes the impact of image distortion on pixels in the first image. Based on this determined distortion parameter and the aforementioned formula, the current feature point corresponding to the predicted feature point can be corrected. This adjusts the current feature point to the position of the predicted feature point, thereby reducing the impact of image distortion on the current feature point corresponding to the predicted feature point.

[0066] Similarly, for each pixel in the first image, after determining the value of the distortion parameter, the pixel in the first image is used as the current feature point in the above formula and substituted into the formula to determine the corrected pixel coordinates of the pixel in the first image, thereby eliminating the influence and obtaining the corrected first image as the second image.

[0067] The determined distortion parameters can also be set at the factory of the shallow depth-of-field camera. As long as the lens and camera of the shallow depth-of-field camera are not disassembled or replaced, the image distortion in the captured image will be automatically corrected when the camera takes an image, and the corrected image will be output.

[0068] In the above technical solution, for each current feature point in the first image, a preset feature point corresponding to that current feature point is determined. The preset feature point is based on the positional relationship between the camera and the calibration board when the first image is captured, mapping feature points on the calibration board to feature points in the first image. Then, based on the current feature point in the first image, the predicted feature point using the camera's undetermined distortion parameters, and the preset feature point, the value of the camera's distortion parameters is determined. Finally, using the determined distortion parameters of the camera, the first image is corrected to obtain the second image. This allows for accurate distortion correction of images captured by the camera.

[0069] Figure 2 shows a schematic flowchart of determining the values ​​of camera distortion parameters based on preset feature points and predicted feature points according to an embodiment of this application. As shown in Figure 2, step S130 may include steps S210 to S230.

[0070] Step S210: Determine the minimization function for the distortion parameters based on the variance of the coordinates between the predicted feature points and the preset feature points corresponding to the predicted feature points.

[0071] For example, there are N current feature points in the first image. For the i-th current feature point, the pixel coordinates (ucorrected,i,vcorrected,i) of the i-th predicted feature point corresponding to the current feature point can be determined according to the above formula, where i≤N. Then, the pixel coordinates (u...) of the i-th predicted feature point corresponding to the i-th preset feature point are determined. i ,v i After that, the minimization function for the distortion parameters can be determined according to the following formula 8:

[0072] Step S220: Perform at least one line search on the minimization function to minimize the function.

[0073] The parameter vector can be determined according to the following formula 9.

[0074] Let the search step size be Δλ k The search direction is The minimum point was found after multiple searches. The predicted feature points obtained at this point are the optimal predicted feature points.

[0075] For example, it can be based on the parameter vector above. The search step size Δλ for each line search is determined by the following formula 10. k :

[0076] Search step size Δλk The selection adopts the line search method, first let It contains the minimum point Among them, for If Δλ exists * ∈[a,b], In [a,Δλ] * Monotonic decline, In [Δλ * The interval [a,b] is monotonically increasing. An initial point Δλ0>0 and an initial step size γ0>0 are selected within the interval [a,b]. The search for γ0 begins from Δλ0. If... Then increase γ0 and continue the search. If in the first step of the search, Then -γ0 is used for the search.

[0077] For example, the search direction can be determined based on the steepest descent direction, Newton's direction, quasi-Newton's direction, conjugate gradient direction, etc.

[0078] Step S230: When the minimization function reaches its minimum value, the current value of the distortion parameter is used as the value of the distortion parameter.

[0079] When the minimization function reaches its minimum value, the pixel coordinates of all predicted feature points are the optimal correction effect of the pixel coordinates of the current feature points in the first image. Therefore, the current value of the determined distortion parameter can make the correction effect of the first image the optimal result.

[0080] In the above technical solution, a minimization function for the distortion parameters is determined based on the variance of the coordinates between the predicted feature points and the corresponding preset feature points. Then, at least one line search is performed on the minimization function to minimize it. When the minimization function reaches its minimum value, the current value of the distortion parameters is used as the value of the distortion parameters. This yields the distortion parameter values ​​that result in the optimal correction effect for the first image.

[0081] Figure 3 shows a schematic flowchart of performing at least one line search on a minimization function according to an embodiment of the present application so that the minimization function obtains its minimum value. As shown in Figure 3, step S220 may include steps S310 to S320.

[0082] In step S310, the search direction for each line search is determined using the quasi-Newton method.

[0083] The search direction can be determined according to the following formula 11.

[0084] in, Indicates the current search direction yes The first derivative (gradient) at the current point, It is the inverse of the approximate Hessian matrix.

[0085] In step S320, the minimization function is subjected to at least one line search according to the determined search direction, and the value of the distortion parameter is updated after each line search until the minimization function obtains the minimum value.

[0086] After determining the search step size and search direction for each line search, each line search can correct the pixel coordinates of the predicted feature point. After multiple searches, the minimum point is found. The predicted feature points obtained at this point are the optimal predicted feature points, and the value of the distortion parameter at this point makes the minimization function reach its minimum value.

[0087] For example, the initial search position can be selected as Where a can be set u and a v All are 1, (u0, v0) represents the coordinates of the image center point, and Δu and Δv are set to 0. We can take the identity matrix as the initial value for the approximate inverse of the Hessian matrix. If the minimum point is found at this point... Then based on the minimum point Corresponding parameter vector Determine the affine transformation matrix. Otherwise, calculate the descent direction. Then along the direction Perform the search and determine the search step size Δλ using Formula 10 above. k >0, let Simultaneously adjust parameters To obtain the corrected For example, the DFP (Davidon-Fletcher-Powell) algorithm and the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm can be used to correct this. If the minimum point is found in this line search... Then based on the minimum point Corresponding parameter vector Determine the value of the distortion parameter. If the minimum point is not found in this line search... Increment the value of k by 1, thereby utilizing the corrected As a parameter The above line search process is repeated cyclically to perform the next line search. Each line search corrects the pixel coordinates of the predicted feature point and determines the value of the distortion parameter after that line search, until the minimum point is determined. This allows us to determine the value of the distortion parameter that minimizes the function.

[0088] In the above technical solution, the quasi-Newton method is used to determine the search direction for each line search. Based on the determined search direction, at least one line search is performed on the minimization function, and the value of the distortion parameter is updated after each line search until the minimization function reaches its minimum value. This allows for a relatively accurate determination of the distortion parameter value that minimizes the function.

[0089] Figure 4 shows a schematic flowchart of performing at least one line search on a minimization function according to another embodiment of the present application so that the minimization function obtains its minimum value.

[0090] As shown in Figure 4, the search range for the distortion parameters, set according to the camera manual and experience, can be determined first. Then, the quasi-Newton method and Formula 11 above are used to determine the search direction for the current sub-line search, and Formula 10 above and the line search method are used to determine the search step size for the current sub-line search. Afterwards, distortion correction can be performed. In this distortion correction, the pixel coordinates of the predicted feature points can be corrected based on the results of the current sub-line search, and the distortion parameter values ​​can be determined at this point. Then, it can be determined whether a minimum point has been found. If a minimum point has been found after the current line search, the distortion parameters can be saved. If a minimum point has not been found after the current line search, the next line search can be performed again starting from the determined search step size.

[0091] Figure 5 shows a schematic flowchart of correcting a first image to obtain a second image using a camera's distortion parameters with predetermined values ​​according to an embodiment of this application. As shown in Figure 5, step S140 may include steps S510 to S520.

[0092] In step S510, the corrected pixels in the first image are determined based on the distortion parameters with the determined values.

[0093] Based on the influence of the distortion parameters with known values ​​on the pixels in the first image, the pixels in the first image that have been repositioned to eliminate the influence can be identified as the corrected pixels in the first image.

[0094] For example, the pixel coordinates in the first image can be used as the current feature point in Formulas 1-7 above, and the corrected pixel in the first image can be used as the predicted feature point in Formulas 1-7 above. Then, the corrected pixel in the first image can be determined based on the distortion parameter with the determined value and Formulas 1-7 above.

[0095] In step S520, a second image is determined based on the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of the pixels adjacent to the corrected pixel.

[0096] Pixel coordinates should all be integers, but the corrected pixel coordinates may not be integers. For example, the corrected pixel coordinates might be (23.7, 45.3). The pixel values ​​at these non-integer positions can be calculated using the pixel values ​​of the neighboring pixels and an interpolation algorithm. The interpolation algorithm can be nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. For high real-time performance, nearest neighbor interpolation or bilinear interpolation can be chosen. For high image quality performance, bicubic interpolation or Lanczos interpolation can be chosen.

[0097] In the above technical solution, the corrected pixels in the first image are determined based on the known distortion parameters. The second image is then determined based on the corrected pixels. Furthermore, for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of its neighboring pixels. This approach takes into account the case where the pixel coordinates of the corrected pixels are not integers, thus obtaining an accurate second image.

[0098] In one possible implementation, a reflector is provided on the flange interface of the camera used to capture the first image. Figure 6 shows a schematic flowchart of whether the optical axis of the camera is perpendicular to the calibration plate according to an embodiment of this application. As shown in Figure 6, the above-described image distortion correction method may further include steps S610 to S620.

[0099] In step S610, the laser autocollimator is used to illuminate the reflector and receive the reflected light signal reflected by the reflector. The laser autocollimator is configured such that the laser emitted by the laser autocollimator is perpendicular to the plane on which the calibration plate is located.

[0100] The basic structure of a laser autocollimator can include a laser source and a mirror, an imaging system, and a detector. The high-precision detector (such as a position-sensitive detector or a charge-coupled device) of the laser autocollimator can accurately measure the offset between the emitted laser and the received reflected signal, thereby determining the angular change. The laser autocollimator emits a laser to illuminate the mirror; if the angle of the mirror changes, the returning laser beam will also deflect. By detecting the offset of the returned beam, the change in the mirror's angle can be calculated.

[0101] In step S620, based on the reflected light signal received by the laser autocollimator, it is determined whether the optical axis of the camera is perpendicular to the calibration plate.

[0102] Figure 7 illustrates a schematic diagram of determining whether the optical axis of a camera is perpendicular to the calibration plate using a laser autocollimator according to an embodiment of this application. As shown in Figure 7, since the reflector is mounted on the camera flange, the Y-axis of the camera coordinate system in Figure 7 can be used to represent the plane where the reflector is located. The Z-axis (optical axis) of the camera coordinate system points to the plane where the calibration plate is located. The laser emitted by the laser autocollimator, perpendicular to the plane where the calibration plate is located, can be used as the incident laser of the reflector. Then, the laser autocollimator can be used to receive the reflected laser returned from the reflector. Based on the angle between the incident laser and the reflected laser, it can be determined whether the optical axis of the camera is perpendicular to the plane where the calibration plate is located, thereby determining whether the optical axis of the camera is perpendicular to the plane where the calibration plate is located.

[0103] For example, whether the camera's optical axis is perpendicular to the calibration plate can be determined using the following formula 12:

[0104] Where ΔY represents the offset between the reflected and incident lasers determined by the laser autocollimator, and L represents the distance between the laser autocollimator and the reflector. When β = 0, it means that the optical axis of the camera is perpendicular to the plane of the calibration plate, that is, the optical axis of the camera is perpendicular to the calibration plate.

[0105] In the above technical solution, a laser autocollimator is used to illuminate a reflector at the camera flange interface and to receive the reflected light signal from the reflector. The laser autocollimator is configured such that the laser emitted by it is perpendicular to the plane of the calibration plate. Then, based on the reflected light signal received by the laser autocollimator, it is determined whether the camera's optical axis is perpendicular to the calibration plate. This allows for a relatively accurate determination of whether the camera's optical axis is perpendicular to the calibration plate, thus determining the camera's posture when capturing the first image if the camera's optical axis is perpendicular to the calibration plate.

[0106] In one possible implementation, the calibration board is configured such that the grid pattern of the calibration board in the first image is arranged parallel to the horizontal direction of the plane in which the first image is located.

[0107] Figure 8 shows a schematic diagram of a first image according to an embodiment of this application. The calibration plate is configured such that the arrangement direction of the checkerboard grid of the calibration plate in the first image is parallel to the horizontal direction of the plane on which the first image is located. As shown in Figure 8, for the plane on which the first image is located, the image coordinate system in which the plane is located can be determined, wherein the direction of the x-axis of the image coordinate system can be taken as the horizontal direction of the plane on which the first image is located. For example, as shown in Figure 8, when the first image is a rectangular image, the upper and lower edges of the first image are parallel to the horizontal direction of the plane on which the first image is located. The area of ​​the calibration plate in the first image contains multiple checkerboard grids, and the arrangement direction of the checkerboard grids is parallel to the horizontal direction of the plane on which the first image is located. In this case, the arrangement direction of the checkerboard grids in the first image can also be parallel to the direction of the upper and lower edges of the first image.

[0108] For example, the calibration plate can be fixed on an optical rotating stage, and the angle of the rotating stage can be adjusted to ensure that the grid pattern of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located.

[0109] The checkerboard pattern in the first image may be bent due to image distortion. However, the image distortion has little impact on the image information near the center of the first image. Therefore, the arrangement direction of the checkerboard pattern in the first image can be used to determine whether the arrangement direction of the checkerboard pattern in the first image is parallel to the horizontal direction of the plane on which the first image is located.

[0110] When the arrangement of the checkerboard grid is parallel to the horizontal direction of the plane where the first image is located, the coordinates of the current feature point in the first image can be determined more easily, avoiding calculation errors that may be caused by the tilt of the checkerboard grid arrangement in the first image.

[0111] Figure 9 shows an exemplary flowchart of image correction using required distortion parameters according to an embodiment of this application. As shown in Figure 9, the above image distortion correction method may further include steps S910 to S920.

[0112] In step S910, the standard deviation between the side length of each chessboard grid and the average side length of each chessboard grid in the calibration board in the second image is determined.

[0113] This standard deviation characterizes the correction effect of the second image. Understandably, the side length of each checkerboard square in the calibration board is the same. Therefore, the side length of each checkerboard square in the ideal image is also the same, and equal to the average side length. Although the pixel coordinates of the feature points in the second image have been corrected, some image distortion may still exist in the second image, resulting in a difference between the second image and the ideal image. Therefore, the side length of each checkerboard square in the second image differs from the average side length of each checkerboard square. The better the correction effect of the second image, the smaller the value of this standard deviation.

[0114] In step S920, when the standard deviation is less than a preset threshold, other images taken by the camera in the same pose are corrected using the distortion parameters with the determined values.

[0115] When the standard deviation is less than the preset threshold, it indicates that the correction effect of the second image meets the requirements. When other images are taken in the same pose using the same camera that took the first image, the distortion parameter values ​​used to correct the first image to the second image can be directly used to correct the image, thereby reducing image distortion in the image.

[0116] In the above technical solution, the standard deviation between the side length of each chessboard grid and the average side length of each chessboard grid in the second image is determined. Then, when the standard deviation is less than a preset threshold, the distortion parameters with the determined values ​​are used to correct other images captured by the camera in the same pose. This avoids the correction effect of the second image being too poor, so that the correction effect can be better when using the determined distortion parameter values ​​to correct other images.

[0117] Figure 10 shows a schematic flowchart of an image distortion correction method according to another embodiment of this application. As shown in Figure 10, a laser autocollimator can be used to correct the camera's verticality so that the camera's optical axis is perpendicular to the calibration plate. Then, an optical rotary stage flattens the calibration plate to ensure that the checkerboard pattern in the ideal image is horizontally aligned with the edge direction of the ideal image. A first image can then be captured using the camera, and the distortion parameter values ​​can be determined based on the feature points in the first image. Then, the corrected pixels in the first image can be determined based on the determined distortion parameter values. A second image is then determined based on the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, image interpolation is performed based on the pixel values ​​of the pixels adjacent to the corrected pixel to determine the pixel value of the corresponding pixel in the second image. Then, an accuracy evaluation can be performed based on the side length of each checkerboard pattern on the calibration plate in the second image and the average side length of each checkerboard pattern to determine the correction effect of the second image. When the accuracy evaluation result is unqualified, the image interpolation and accuracy evaluation are repeated starting from the determined distortion parameter values. When the accuracy evaluation result is satisfactory, a second image can be output as a correction image.

[0118] Figure 11 shows a schematic block diagram of an image distortion correction apparatus according to an embodiment of the present application. As shown in Figure 11, the image distortion correction apparatus provided according to a second aspect of the present application includes an image receiving module 1110, a feature point matching module 1120, a parameter determination module 1130, and an image correction module 1140.

[0119] The image receiving module 1110 is used to acquire the first image of the calibration board taken by the camera.

[0120] The feature point matching module 1120 is used to determine the current feature point in the first image corresponding to each preset feature point. The preset feature points are based on the positional relationship between the camera and the calibration board when the camera captures the first image, and the feature points on the calibration board are mapped to the feature points in the image captured by the camera.

[0121] The parameter determination module 1130 is used to determine the values ​​of the camera's distortion parameters based on preset feature points and predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the undetermined distortion parameters of the camera.

[0122] The image correction module 1140 is used to correct the first image using the camera's known distortion parameters to obtain the second image.

[0123] In one possible implementation, the parameter determination module 1130 includes a function calculation submodule, a line search submodule, and a parameter value determination submodule. The function calculation submodule determines a minimization function for the distortion parameters based on the variance of the coordinates between the predicted feature point and a preset feature point corresponding to that predicted feature point. The line search submodule performs at least one line search on the minimization function to minimize the function. The parameter value determination submodule uses the current value of the distortion parameters as the value of the distortion parameters when the minimization function reaches its minimum value.

[0124] In one possible implementation, the line search submodule includes a search direction determination submodule and a minimum value determination submodule. The search direction determination submodule is used to determine the search direction for each line search using a quasi-Newton method. The minimum value determination submodule is used to perform at least one line search on the minimization function according to the determined search direction, and update the value of the distortion parameter after each line search, until the minimization function obtains the minimum value.

[0125] In one possible implementation, the image correction module 1140 includes a pixel determination submodule and a second image determination submodule. The pixel determination submodule is used to determine corrected pixels in the first image based on a distortion parameter with a predetermined value. The second image determination submodule is used to determine a second image based on the corrected pixels, wherein, for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of the pixels adjacent to the corrected pixel.

[0126] In one possible implementation, the first image is taken using the camera with the camera's optical axis perpendicular to the calibration plate.

[0127] In one possible implementation, the camera's flange interface is equipped with a reflector, and the image distortion correction device further includes a laser autocollimator control submodule and a perpendicularity determination module. The laser autocollimator control submodule is used to illuminate the reflector with a laser autocollimator and receive the reflected light signal from the reflector, wherein the laser autocollimator is configured such that the laser emitted by the laser autocollimator is perpendicular to the plane of the calibration plate. The perpendicularity determination module is used to determine whether the camera's optical axis is perpendicular to the calibration plate based on the reflected light signal received by the laser autocollimator.

[0128] In one possible implementation, the calibration board is configured such that the grid pattern of the calibration board in the first image is arranged parallel to the horizontal direction of the plane in which the first image is located.

[0129] In one possible implementation, the image distortion correction device further includes a standard deviation determination submodule and a distortion correction submodule. The standard deviation determination submodule is used to determine the standard deviation between the side length of each checkerboard grid and the average value of the side lengths of each checkerboard grid in the second image. The distortion correction submodule is used to correct other images captured by the camera in the same pose using the determined distortion parameters when the standard deviation is less than a preset threshold.

[0130] In one possible implementation, the unknown distortion parameters include the camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

[0131] According to a third aspect of this application, an electronic device is also provided. FIG12 shows a schematic block diagram of an electronic device according to an embodiment of this application. As shown in FIG12, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the image distortion correction method as described above.

[0132] Furthermore, according to a fourth aspect of this application, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the image distortion correction method described in the embodiments of this application, and is used to implement corresponding modules in the image distortion correction apparatus described in the embodiments of this application. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0133] According to a fifth aspect of this application, a computer program product is also provided, including computer program instructions that, when executed, perform the image distortion correction method described above.

[0134] Those skilled in the art can understand the specific implementation and beneficial effects of the above-described image distortion correction device, electronic device, storage medium, and computer program product by reading the detailed description of the image distortion correction method above, and for the sake of brevity, they will not be described in detail here.

[0135] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0138] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0139] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0140] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0141] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0142] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the image distortion correction apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0143] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0144] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. An image distortion correction method, characterized in that, The method includes: Acquire the first image of the calibration board captured by the camera; For each current feature point in the first image, a preset feature point corresponding to the current feature point is determined. The preset feature point is based on the positional relationship between the camera and the calibration board when the camera captures the first image, and the feature points on the calibration board are mapped to the feature points in the first image. The distortion parameters of the camera are determined based on the preset feature points and the predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the undetermined distortion parameters of the camera. The first image is corrected using the camera's known distortion parameters to obtain the second image.

2. The method according to claim 1, characterized in that, The step of determining the values ​​of the camera's distortion parameters based on the preset feature points and predicted feature points includes: Based on the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point, a minimization function for the distortion parameter is determined; Perform at least one line search on the minimization function to minimize the function; When the minimization function reaches the minimum value, the current value of the distortion parameter is used as the value of the distortion parameter.

3. The method according to claim 2, characterized in that, Performing at least one line search on the minimization function to minimize the function includes: The search direction for each line search is determined using a quasi-Newton method; The minimization function is subjected to at least one line search according to the determined search direction, and the value of the distortion parameter is updated after each line search until the minimization function obtains the minimum value.

4. The method according to any one of claims 1 to 3, characterized in that, The step of correcting the first image using the camera's determined distortion parameters to obtain the second image includes: Based on the distortion parameters with the determined values, the corrected pixels in the first image are determined; Based on the corrected pixels, the second image is determined, wherein for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of the pixels adjacent to the corrected pixel.

5. The method according to any one of claims 1 to 3, characterized in that, The first image was taken using the camera with the camera's optical axis perpendicular to the calibration plate.

6. The method according to claim 5, characterized in that, The camera's flange interface is equipped with a reflector. The method further includes: The reflector is illuminated by a laser autocollimator, and the reflected light signal reflected from the reflector is received. The laser autocollimator is configured such that the laser emitted by the laser autocollimator is perpendicular to the plane on which the calibration plate is located. Based on the reflected light signal received by the laser autocollimator, it is determined whether the optical axis of the camera is perpendicular to the calibration plate.

7. The method according to any one of claims 1 to 3, characterized in that, The calibration board is configured such that the grid pattern of the calibration board in the first image is parallel to the horizontal direction of the plane containing the first image.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Determine the standard deviation between the side length of each chessboard square in the calibration board in the second image and the average of the side lengths of each chessboard square; When the standard deviation is less than a preset threshold, the distortion parameter with the determined value is used to correct other images taken by the camera in the same pose.

9. The method according to any one of claims 1 to 3, characterized in that, The unknown distortion parameters include the camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

10. An image distortion correction device, characterized in that, include: The image receiving module is used to acquire the first image of the calibration board captured by the camera; The feature point matching module is used to determine the current feature point in the first image corresponding to each preset feature point. The preset feature point is based on the positional relationship between the camera and the calibration board when the camera captures the first image, and maps the feature points on the calibration board to the feature points in the image captured by the camera. The parameter determination module is used to determine the value of the distortion parameter of the camera based on the preset feature points and the predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the undetermined distortion parameters of the camera. An image correction module is used to correct the first image using the camera's pre-determined distortion parameters to obtain a second image.

11. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the image distortion correction method as described in any one of claims 1 to 9.

12. A storage medium on which program instructions are stored, characterized in that, The program instructions are used to execute the image distortion correction method as described in any one of claims 1 to 9 when the program is run.

13. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the image distortion correction method as described in any one of claims 1 to 9.

Images