Apparatus for detecting change in camera installation state and correcting camera setting

Abstract

An apparatus for correcting a setting of a pre-installed camera is proposed. The apparatus may include: a memory storing features of at least one feature point, image coordinates, measured GPS coordinates, measured conversion GPS coordinates, and a homography matrix; and a processor configured to obtain an image, and update the homography matrix or the feature point according to detection of a change in the feature point in the obtained image.

Description

Device for detecting changes in camera installation status and calibrating camera settings

[0001] The present invention relates to an apparatus for detecting a change in camera installation status and correcting camera settings, and more specifically, to a technology for detecting a change in camera installation status through an acquired image and performing camera setting correction accordingly.

[0002] FIG. 1 illustrates a traffic safety service system based on communication technology, which is one of the fields to which the present invention is applied. For example, V2X (vehicle to everything) and Soft V2X technologies are used.

[0003] The RSU (roadside unit) (100) transmits road and traffic information provided by the C-ITS (Cooperative Intelligent Transport System) (200) to road users. Additionally, the RSU (100) collects information about surrounding vehicles. Furthermore, the RSU (100) links C-ITS and Soft V2X services using different technologies and utilizes various roadside sensor information, including CCTVs, to provide information that can predict collision accidents on the road.

[0004] Additionally, the RSU (100) can detect various road users, such as pedestrians, vehicles, motorcycles, and kickboards, using intelligent CCTV. The RSU (100) transmits road user detection information to nearby users to enable collision prediction or avoidance. The RSU (100) can transmit road user detection information to a C-ITS terminal (311) and a Soft V2X terminal (321). The C-ITS terminal (311) is installed in a vehicle (310), and the Soft V2X terminal (321) is shown installed or located in a vehicle (320).

[0005] As such, C-ITS and Soft V2X are technologies that help with collision prediction, avoidance, or traffic accident prevention, respectively, but the C-ITS terminal (311) and the Soft V2X terminal (321) cannot directly exchange information with each other. In this way, by linking C-ITS and Soft V2X through the RSU (100), information exchange between the C-ITS terminal and the Soft V2X terminal is made possible, thereby enabling collision prediction and avoidance between the C-ITS terminal user and the Soft V2X terminal user.

[0006] Soft V2X is a cloud-based service that collects or analyzes the location, direction, and speed of pedestrians and vehicles to notify users of potential traffic safety risks in real time. For such a service, camera installation is critical, but it is a very cumbersome task due to the complex procedures involved. In particular, if the installation location is shifted due to weather (wind / rain) or birds (pigeons), complex procedures must be reapplied from scratch to calibrate the cameras. This invention aims to resolve these complex procedures and propose a method for automatically calibrating cameras.

[0007] The present invention proposes a device for detecting changes in camera installation status and correcting camera settings.

[0008] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0009] According to one embodiment of the present invention, a device for setting correction of a pre-installed camera is proposed, and may include a memory storing at least one feature point feature, image coordinates, actual GPS coordinates, actual converted GPS coordinates, and a homography matrix; and a processor configured to acquire an image and, upon detection of a change in the feature point in the acquired image, update the homography matrix or update the feature point.

[0010] Additionally or alternatively, the processor may be configured to acquire motion vectors of at least some of the feature points in the acquired image and to update the homography matrix based on the acquired motion vectors, and the motion vectors may be configured to be acquired based on the position of the feature points on the reference image and the position of the feature points on the acquired image.

[0011] Additionally or alternatively, the processor may be configured to acquire motion vectors of feature points that do not overlap with the area (bbox) for the object detected in the acquired image.

[0012] Additionally or alternatively, the processor may be configured to divide the area of ​​the acquired image into four sections based on the centroid of a feature point that does not overlap with the area (bbox) for the object detected in the acquired image, select a preset number of feature points in each of the four divided areas of the acquired image, and acquire the motion vector of the selected feature points.

[0013] Additionally or alternatively, the processor may be configured to periodically acquire images, check whether the distribution of the feature points in the acquired images is uniform, calculate a change value of the feature points in the periodically acquired images based on whether the distribution of the feature points is uniform, and acquire an update time of the homography matrix based on the calculated change value.

[0014] Additionally or alternatively, the change value of the feature point may be configured to include at least one of the average value or median value of the motion vector of the feature point, the maximum value of the motion vector of the feature point, or the change in the number of the feature points.

[0015] Additionally or alternatively, the processor may be configured to output to a human-machine interface (HMI) the time when the homography matrix needs to be updated or the time when the settings of the installed camera need to be checked.

[0016] Additionally or alternatively, the processor may be configured to output a user interface component corresponding to a change value of the feature point to a human-machine interface (HMI).

[0017] Additionally or alternatively, the processor may be configured to output a user interface component corresponding to the change value of the feature point to a human-machine interface (HMI) together with at least two of the periodically acquired images.

[0018] Additionally or alternatively, the processor may be configured to acquire estimated GPS information of at least one object detected based on the homography matrix, calculate the difference between the estimated GPS information and the actual GPS information of at least one object detected, and determine whether to perform calibration of the installed camera or update the homography matrix based on the calculated difference.

[0019] Additionally or alternatively, the processor may be configured to update the homography matrix using the changed image coordinates of the feature points as the change in image coordinates of at least some of the feature points in the acquired image exceeds a threshold range.

[0020] Additionally or alternatively, the processor may be configured to set a window for searching for feature points in the acquired image, detect features of the stored feature points in the set window, and acquire changed image coordinates of the feature points corresponding to the detected features.

[0021] Additionally or alternatively, the homography matrix may be configured to be updated using the changed image coordinates of the feature points and the actual GPS coordinates or actual transformed GPS coordinates of the feature points.

[0022] Additionally or alternatively, among the stored feature points, feature points for which no feature was detected in the acquired image may be configured not to be used in updating the homography matrix.

[0023] Additionally or alternatively, the processor may be configured to update features for the stored feature points after the homography matrix is ​​updated, and to update the actual transformed GPS coordinates of the stored feature points using the updated homography matrix.

[0024] Additionally or alternatively, the processor may be configured to determine to update the homography matrix or to determine that the updated homography matrix is ​​valid by using at least one of the number of feature points detected in the acquired image, the distance between the feature points in the acquired image, or the difference between the actual transformed GPS coordinates of the feature points and the actual GPS coordinates according to the updated homography matrix and the changed image coordinates of the feature points.

[0025] Additionally or alternatively, the processor may be configured to determine to update the homography matrix or determine that the updated homography matrix is ​​valid if the number of feature points detected in the acquired image is greater than or equal to a preset number, the distance between the preset number of feature points in the acquired image is within a preset distance, and the difference between the actual transformed GPS coordinates of the preset number of feature points acquired using the updated homography matrix and the changed image coordinates and the actual GPS coordinates is within a preset range.

[0026] Additionally or alternatively, the processor may be configured to update the feature points as at least some of the feature points are not detected in the acquired image.

[0027] Additionally or alternatively, the processor is configured to acquire a new feature point in the acquired image through a corner detection algorithm, acquire actual transformed GPS coordinates of the new feature point using the homography matrix, and acquire features of the new feature point, and the acquired new feature point may be configured to be located in an area outside a pre-specified exclusion area in the acquired image.

[0028] According to one embodiment of the present invention, a method for correcting the settings of a camera that has been installed is proposed, and the method is performed by a device for correcting camera settings and may be configured to include the step of acquiring an image; and the step of updating a previously stored homography matrix or updating the feature points according to the detection of a change in feature points in the acquired image.

[0029] The above-mentioned problem-solving methods are merely some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by those skilled in the art based on the detailed description of the present invention to be described below.

[0030] The present invention has the following effects.

[0031] The present invention can maintain the accuracy of estimating the position of objects within an image obtained through a camera by correcting camera settings using an acquired image without physically reinstalling the camera when the installation location of the camera is changed.

[0032] In addition, the present invention can maintain the accuracy of the position estimation of objects within an image acquired through a camera by performing camera settings using an acquired image when the surrounding environment in which the camera is installed changes.

[0033] The effects according to the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the following detailed description of the invention.

[0034] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and explain the technical concept of the present invention together with the detailed description.

[0035] FIG. 1 illustrates a system diagram of a field to which the present invention is applied.

[0036] FIG. 2 illustrates a flowchart for camera-based object tracking and position estimation to which the present invention is applied.

[0037] Figure 3 illustrates a pinhole camera model for explaining camera calibration.

[0038] Figure 4 illustrates the conversion between image coordinates and GPS coordinates using a homography matrix according to the present invention.

[0039] FIG. 5 illustrates a flowchart for the setting process of a camera according to the present invention.

[0040] FIG. 6 illustrates a flowchart of a process for detecting changes in feature points in an acquired image according to the present invention.

[0041] FIG. 7 illustrates a flowchart of a process for correcting camera settings according to feature point changes in an acquired image according to the present invention.

[0042] FIG. 8 illustrates a flowchart of a process for determining whether to initiate camera setting correction or the validity of camera setting correction based on the state of feature points of an acquired image according to the present invention.

[0043] FIG. 9 illustrates a flowchart of a process for setting new feature points according to feature point changes in an acquired image according to the present invention.

[0044] FIG. 10 shows an area excluded from feature point setting in an image obtained according to the present invention.

[0045] FIG. 11 illustrates a flowchart of a method for setting correction of a pre-installed camera according to the present invention.

[0046] FIG. 12a shows an area for feature points and detected objects displayed on an acquired image according to the present invention, and FIG. 12b shows an area for feature points and detected objects with the original content of the acquired image removed.

[0047] FIG. 13a shows an image divided into four sections based on the center of gravity of a plurality of feature points according to the present invention, and FIG. 13b shows a feature point selected in each of the four divided regions.

[0048] FIG. 14 shows an image with motion vectors displayed according to the present invention.

[0049] FIG. 15 illustrates a compensation or update operation according to the movement of a camera according to the present invention.

[0050] FIG. 16 illustrates a flowchart of a method for obtaining the update time of a homography matrix according to the present invention.

[0051] Figure 17 shows the update time of the homography matrix according to the magnitude of the change value according to the present invention.

[0052] FIGS. 18 to 21 illustrate a user interface according to the present invention.

[0053] FIG. 22 illustrates a flowchart of a method for setting correction of an installed camera based on GPS (global positioning system) tracking according to the present invention.

[0054] FIG. 23 illustrates a block diagram of a device for setting correction of a camera according to the present invention.

[0055] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0056] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0057] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0058] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0059] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0060]

[0061] FIG. 2 illustrates a procedure in a camera-based object tracking and position estimation system to which the present invention is applied.

[0062] First, an image is acquired using an image sensor such as a camera (S10).

[0063] Object detection is performed on the acquired image (S20).

[0064] Then, tracking of the detected object is performed (S30).

[0065] Finally, position estimation is performed to obtain position information, such as GPS coordinates, from the location of an object in the acquired image (S40).

[0066] Object tracking uses computer vision technology to detect and track objects in images. To this end, various algorithms are used to detect and track moving objects. Major algorithms include, for example, the Kalman filter, the particle filter, and DeepSORT. An identifier (ID) is assigned to each tracked object to track them individually. This allows for the estimation of each object's movement path and velocity.

[0067]

[0068] Figure 3 illustrates a pinhole camera model for explaining camera calibration.

[0069] Camera calibration is the process of adjusting a camera so that it can correctly capture and interpret images. This process improves image accuracy by correcting lens distortion, color distortion, and geometric distortion. The primary goal is to accurately reproduce real-world objects, and this plays a crucial role in various fields such as computer vision, robotics, augmented and virtual reality, and autonomous vehicles.

[0070] Referring to Fig. 3, a total of four coordinate systems are shown, starting from the bottom right: the camera coordinate system, the normalized coordinate system, the pixel coordinate system, and the world coordinate system.

[0071] The world coordinate system and the camera coordinate system are 3D coordinate systems, while the normalized coordinate system and the pixel coordinate system are 2D coordinate systems. The world coordinate system is the coordinate system we use as a reference when expressing the position of objects, and the camera coordinate system is a coordinate system based on the camera. The pixel coordinate system is also referred to as the image coordinate system; it is the coordinate system for the image we actually see with our eyes, and it is described as an image plane formed with the top-left corner as the origin and extending in the right and downward directions.

[0072] A point P=(X,Y,Z) in 3D space is projected onto a point pimg = (x, y) in the image plane through the focal point of a camera or lens.

[0073] The normalized coordinate system corresponds to an image coordinate system from which the influence of the camera's intrinsic parameters has been removed. It is a coordinate system with units removed (normalized) and defines a virtual image plane where the distance from the camera focal point is 1. It is defined as a point P'=(u,v) on the normalized coordinate system. Accordingly, if the camera's intrinsic parameters are known, it is possible to convert between image (pixel) coordinates and normalized coordinates.

[0074]

[0075] Here, fx is the horizontal focal length, fy is the vertical focal length, and cx and cy are principal points representing the image (pixel) coordinates where the optical axis and the image plane intersect.

[0076] In addition, a rotation and translation transformation matrix ([R│t]) for converting the world coordinate system to the camera coordinate system is modeled, which can be expressed as follows.

[0077]

[0078] From a mathematical perspective, camera calibration is the process of finding the transformation relationship between 3D spatial coordinates and 2D image coordinates as described above, or the parameters that explain this transformation relationship.

[0079] The matrix in Equation 1 is referred to as the inner parameter, and [R│t] in Equation 2 is referred to as the outer parameter.

[0080] Using wide-angle or ultra-wide-angle lenses with a wide camera field of view allows you to see a wide range, but this leads to the problem of relatively severe image distortion.

[0081] In addition to visual issues, such image distortion poses a particular problem when accurate numerical calculations are required through image analysis. For instance, if image coordinates are converted to physical coordinates to determine the actual location of an object detected in an image, significant errors will occur depending on the degree of image distortion.

[0082] Distortion caused by a camera, or lens, includes radial distortion and tangential distortion.

[0083] Assuming there is no lens distortion, a point (Xc, Yc, Zc) in 3D space is projected onto a point (x_n_u, y_n_u) on a normalized image plane by shim projection.

[0084]

[0085] If the normalized coordinates reflecting lens distortion are expressed as (x_n_d, y_n_d), the lens distortion model can be expressed as follows.

[0086]

[0087] Here, am.

[0088] In mathematical equation 4, the first term on the right side represents radial distortion, and the second term represents tangential distortion. K1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients, and r_u is the distance (radius) to the center (principal point) when there is no distortion.

[0089] Again, using the intrinsic parameters of Equation 1, the normalized coordinates with distortion can be represented as image pixel coordinates (x_p_d, y_p_d).

[0090]

[0091] Here, skew_cfx is an asymmetry coefficient, usually set to 0 (i.e., omitted).

[0092] In other words, coordinates on a normalized image plane can be converted into coordinates that reflect distortion on the image plane.

[0093] Through such relationships, image distortion correction or image coordinate distortion correction can be performed, and image coordinate distortion correction is explained.

[0094] In actual image processing, obtaining the corrected image coordinates can be much more important than obtaining the distorted image itself.

[0095] However, finding distortion-corrected image coordinates from distorted image coordinates is a difficult problem with no closed-form solution, so an approximate solution is usually obtained.

[0096] One method to obtain the distortion-corrected coordinate p_u from the distorted image coordinate p_d is to assume p_u = p_d and distort p_u using Equation 4. The error between the point obtained in this way and the actual p_d is calculated, and that error is inversely reflected in p_u. For example, if the point where p_u is distorted is greater than p_d by δ, p_u is reduced by δ. This process is repeated until the error becomes less than or equal to a threshold value.

[0097] In conclusion, camera calibration or camera distortion correction can calculate intrinsic or extrinsic parameters using pairs of image (pixel) coordinates and corresponding 3D coordinates (i.e., world coordinates), and correct distortion of image coordinates based thereon.

[0098] Although camera calibration and camera distortion correction have been briefly described in this specification, reference may be made to the many materials disclosed prior to the filing of this application.

[0099]

[0100] Figure 4 illustrates the conversion between image coordinates and GPS coordinates using a homography matrix according to the present invention.

[0101] Figure 4 (a) shows the conversion of image (video) coordinates to GPS coordinates through a homography matrix, and Figure 4 (b) shows the conversion of GPS coordinates to image coordinates through a homography matrix.

[0102] In image-based position recognition or mapping technology, a homography matrix can be used to perform alignment between world (real-world) coordinates and image coordinates. A homography matrix is ​​a 3×3 matrix representing the correspondence between two planes and defines the transformation between a point on a real-world plane and a point in an image.

[0103] For example, when the world coordinate system is expressed as a specific planar projection coordinate system on Earth, such as Universal Transverse Mercator (UTM) coordinates, a point on a specific plane (e.g., road surface, building exterior wall, etc.) in the UTM coordinate system can be mutually transformed with the 2D coordinates within an image acquired by a camera through a homography matrix. In this case, the relationship between the world coordinates (X,Y,1)^T and the image coordinates (x,y,1)^T is expressed as follows.

[0104]

[0105] Here, H is a 3×3 homography matrix, and the above relationship represents a scale proportional relationship in a homogeneous coordinate system.

[0106] In addition, since it is difficult to directly correspond one-to-one with image coordinates using latitude and longitude coordinate systems such as GPS coordinates, GPS coordinates are usually converted to a planar coordinate system such as UTM coordinates, and the transformation relationship between that coordinate system and the image coordinates is expressed as a homography matrix. Therefore, through the procedures of converting GPS coordinates to UTM coordinates and converting UTM coordinates to image coordinates (applying homography), absolute coordinates based on a satellite positioning system and relative coordinates based on images can be integratedly mapped.

[0107]

[0108] FIG. 5 illustrates a flowchart for the setting process of a camera according to the present invention.

[0109] FIG. 5 illustrates the process of setting (i.e., calibration) a camera after a camera such as a CCTV has been installed. The process of FIG. 5 is performed by a device (10) for correcting camera settings, and below, it will be simply described as being performed by the device (10).

[0110] The device (10) can be configured to obtain GPS coordinates of a specific point, i.e., a point within the range captured by the camera (S510).

[0111] The device (10) can be configured to acquire image coordinates of the specific point (S520).

[0112] Then, the device (10) may be configured to pair the GPS coordinates of the specific point with the image coordinates (S530). Additionally, the device (10) is configured to assume the acquired GPS coordinates as 3D coordinates on a plane, set one of the measured points as p0 (origin), set the x-axis and y-axis coordinates of the 3D coordinates to the longitude and latitude coordinates of the GPS coordinates, respectively, and set the z-axis coordinate to 0. After that, the device (10) is configured to express the 3D coordinates of the remaining points as coordinates relative to p0, and set the z-axis coordinate to 0. The 3D coordinates acquired in this specification are referred to as “actually converted GPS coordinates.”

[0113] Additionally, the device (10) may be configured to extract features of the specific point and store features of the specific point (S540).

[0114] The device (10) can be configured to perform camera settings (i.e., calibration) using acquired GPS coordinates and acquired image coordinates (S550). Refer to FIG. 3 for camera calibration. The difference between the description of FIG. 3 and the calibration of S550 is that the previously acquired actual converted GPS coordinates are used for calibration instead of the world coordinate system.

[0115] The device (10) can be configured to calculate a homography matrix using a pair of acquired image coordinates and actual converted GPS coordinates (S560).

[0116]

[0117] FIG. 6 illustrates a flowchart of a process for detecting changes in feature points in an acquired image according to the present invention. The process of FIG. 6 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0118] The device (10) may be configured to acquire an image (S610). The device (10) may be configured to detect features in the acquired image based on the features of a specific point mentioned in the description related to FIG. 5.

[0119] The device (10) may be configured to determine whether feature points have changed in the acquired image (S620).

[0120] The device (10) can terminate the process if the feature points in the acquired image have not changed.

[0121] The device (10) can be configured to determine whether the image coordinates have changed when it is determined that the feature points in the acquired image have changed (S630).

[0122] In the case where the image coordinates have changed, the device (10) can be configured to update the homography matrix based on the changed image coordinates (S640).

[0123] Whether a case in which image coordinates have changed can be determined based on whether the image coordinates of a preset number of feature points among the preset feature points have changed. For example, if there are N preset feature points and the image coordinates of n (n <= N) feature points have changed, the device (10) can be configured to determine that it is a case in which image coordinates have changed.

[0124] As it is not a case where the image coordinates have changed, the device (10) may be configured to generate or update new feature points in the acquired image (S650). A case where the image coordinates have not changed means a case where a pre-set feature point is no longer found in the acquired image. For example, if there are N pre-set feature points and n (n <= N) feature points cannot be found in the acquired image, the device (10) may be configured to determine that it is not a case where the image coordinates have changed (i.e., a case where the feature point has disappeared).

[0125]

[0126] FIG. 7 illustrates a flowchart of a process for correcting camera settings according to a change in feature points in an acquired image according to the present invention. The process of FIG. 7 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0127] Figure 7 is a diagram specifically explaining S640 of Figure 6.

[0128] The device (10) can be configured to set a window for navigation in the acquired image (S641).

[0129] The device (10) may be configured to detect a pre-set feature point in a set window and to obtain the image coordinates of the detected feature point (S642). The pre-set feature point detected in S642 refers to a feature point used in advance during camera calibration, and the device (10) may detect the feature point through a feature matching technique. Accordingly, the device (10) may be configured to retrieve the features of the pre-set feature point stored in memory, etc.

[0130] The device (10) may be configured to perform camera calibration using the image coordinates of the detected feature point (S643). More specifically, the device (10) may be configured to perform camera calibration using the image coordinates of the detected feature point and the previously acquired actual converted GPS coordinates of the detected feature point. Since FIG. 7 is a case where the image coordinates of the feature point have been changed, the actual location of the feature point (i.e., GPS coordinates) has not been changed, so camera calibration is performed using the changed image coordinates of the feature point and the previously acquired actual converted GPS coordinates.

[0131] Additionally, the device (10) may be configured to calculate a homography matrix using pairs of acquired image coordinates and actual converted GPS coordinates of the detected feature points (S644).

[0132] Then, the device (10) can be configured to update the features of the detected feature points, image coordinates, actual converted GPS coordinates, etc. (S645).

[0133]

[0134] FIG. 8 illustrates a flowchart of a process for determining whether to initiate camera setting correction or the validity of camera setting correction based on the state of feature points of an acquired image according to the present invention. The process of FIG. 8 is performed by a device (10) for camera setting correction, and below, it will be described simply as being performed by the device (10).

[0135] The device (10) may be configured to determine whether the number of feature points detected in the acquired image is greater than or equal to a preset number (S810). If the number of detected feature points is less than the preset number, the process may be terminated.

[0136] Additionally, the device (10) may be configured to determine whether the distance between detected feature points is within a preset distance, depending on whether the number of feature points detected in the acquired image is greater than or equal to a preset number (S820). In this case, the average distance between the detected feature points may be compared with the preset distance. If the distance between the detected feature points exceeds the preset distance, the process may be terminated.

[0137] Additionally, the device (10) may be configured to determine whether the error between the actual converted GPS coordinates of the detected feature points and the actual GPS coordinates is within a preset range, depending on whether the distance between the detected feature points is within a preset distance (S830). The actual converted GPS coordinates of the detected feature points can be obtained by using the homography matrix and the image coordinates of the detected feature points. If the error between the actual converted GPS coordinates of the detected feature points and the actual GPS coordinates exceeds a preset range, the process may be terminated.

[0138] The device (10) may be configured to determine whether to update the homography matrix or to determine that the updated homography matrix is ​​valid if the error between the actual converted GPS coordinates of the detected feature point and the actual GPS coordinates is within a preset range.

[0139] Accordingly, the process of FIG. 8 may be used as a preliminary process to initiate S640 of FIG. 6, or as a post-process to determine whether the updated homography matrix is ​​valid after completing S640 of FIG. 6 or after the procedure of FIG. 7 is completed.

[0140]

[0141] FIG. 9 illustrates a flowchart of a process for setting new feature points according to feature point changes in an acquired image according to the present invention. The process of FIG. 9 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0142] The device (10) can be configured to acquire a new feature point in the acquired image (S651). FIG. 9 is for a case where the image coordinates of the feature point in S630 of FIG. 6 are not changed, that is, a case where no more feature points can be found in the acquired image, so it is necessary to set a new feature point.

[0143] The acquisition of new feature points can be performed using feature extraction algorithms such as corner detection algorithms.

[0144] The device (10) may be configured to filter acquired feature points (S652). This may be done depending on whether there is an exclusion area set in the acquired image. If an exclusion area is set, the acquired feature points in the exclusion area may not be used.

[0145] The device (10) can be configured to obtain GPS coordinates of a new feature point using a homography matrix (S653).

[0146] Then, the device (10) can be configured to update the features of the new feature point (S654). The device (10) can be configured to pair the GPS coordinates and image coordinates of the new feature point and store them in a storage medium such as memory.

[0147]

[0148] FIG. 10 shows an area excluded from feature point setting in an image obtained according to the present invention.

[0149] FIG. 10(a) shows an image acquired by a camera. FIG. 10(b) shows an exclusion region (DC; Don't Care, OC; occlusion). Feature points located in the exclusion region may be excluded from the feature point setting for the present invention.

[0150]

[0151] FIG. 11 illustrates a flowchart of a method for correcting the settings of a pre-installed camera according to the present invention. The process of FIG. 11 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0152] The device (10) may be configured to load an image (S1101). The image may be a picture of the surrounding road conditions, as shown in FIG. 10.

[0153] The device (10) may be configured to specify a feature point area (S1102). The feature point area may refer to a polygonal or circular area of ​​a preset size based on the feature point.

[0154] (Illegible) and (Illegible)

[0155] The device (10) can be configured to filter feature point regions according to preset conditions (S1103).

[0156] Here, the pre-set condition may include the overlapping of the bounding box (bbox) for the object detected in the image, that is, the area for the rectangular object, and the feature point area. If the area for the detected object and the feature point area overlap, the device (20) may exclude the corresponding feature point area or feature point from the method for correcting camera settings according to the present invention.

[0157] The device (10) may be configured to select some of the multiple feature points (S1104).

[0158] More specifically, the device (10) may be configured to divide the image into four sections based on the center of gravity of a plurality of feature points.

[0159] The device (10) may be configured to divide the area of ​​the acquired image into four sections based on the centroid of a feature point that does not overlap with the area (bbox) for the object detected in the acquired image. Referring to FIG. 13a, two straight lines are shown that divide the image illustrated in FIG. 12a into four sections. The intersection point of the two straight lines corresponds to the centroid of a plurality of feature points.

[0160] Additionally, the device (10) may be configured to select a preset number of feature points in each of the four divided regions of the acquired image. Referring to FIG. 13b, it can be seen that two straight lines dividing the image into four and a preset number of feature points selected in each of the four divided regions are displayed.

[0161] The device (10) may be configured to acquire a motion vector of a selected feature point (S1105). The motion vector represents a change in the position of a feature point in the acquired image and a feature point in the “reference image” corresponding to this feature point. Alternatively, the motion vector may be acquired based on the position of the feature point on the reference image and the position of the feature point on the acquired image. That is, the position of the feature point at the time the reference image is acquired is compared with the position of the feature point at the time the image loaded in S1101 is acquired. Through this, the motion vector can indicate how much the position or PTZ of the camera capturing the image has changed. Referring to FIG. 14, it can be seen that a motion vector represented by an arrow is displayed on the image.

[0162] The device (10) may be configured to update the homography matrix based on motion vectors (S1106). The device (10) may be configured to update the homography matrix based on changes in the position of identical or corresponding feature points across two images.

[0163] Figure 15 (a) shows motion compensation when translation motion of the camera occurs, and Figure 15 (b) shows the update of the homography matrix when rotation motion of the camera occurs. In the case of translation, a correction matrix corresponding to the average value of the motion vector is added; in the case of rotation, the homography matrix is ​​updated based on the motion vector to perform compensation for camera movement or distortion.

[0164]

[0165] FIG. 16 illustrates a flowchart of a method for obtaining an update time of a homography matrix according to the present invention. The process of FIG. 16 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0166] The device (10) may be configured to acquire an image (S1601). The acquisition of the image may be performed periodically.

[0167] The device (10) may be configured to determine whether the number of matching feature points is smaller than a threshold (S1602).

[0168] A camera according to the present invention is installed at a specific point and configured to photograph a pre-designated area, and accordingly, information regarding feature points detected on an image acquired through the camera may be stored in advance. In this situation, the device (10) may be configured to search for feature points in a newly acquired image. If the number of feature points found in the newly acquired image is too small, that is, if the number of feature points matching (with feature points of a previously acquired image or reference image) is smaller than a threshold, this means that the camera's position has changed or been misaligned relatively excessively, so the device (10) may be configured to activate an emergency recovery mode (S1606).

[0169] Based on the number of matching feature points being greater than or equal to a threshold, the device (10) may be configured to determine whether the distribution of feature points is uniform (S1603). The distribution of feature points may be determined based on the location of the feature points on the acquired image. Whether the distribution of feature points is uniform may be determined by whether the feature points are not concentrated in a specific area on the image. To this end, the device (10) may be configured to uniformly divide the entire area of ​​the image and check the number of feature points belonging to each divided area. Based on the deviation of the number of feature points in each divided area exceeding a threshold, the device (10) is configured to determine that the distribution of feature points is not uniform; based on the deviation of the number of feature points in each divided area being less than or equal to a threshold, the device (10) is configured to determine that the distribution of feature points is uniform.

[0170] The device (10) can terminate the procedure illustrated in FIG. 16 based on the fact that the distribution of feature points is not uniform.

[0171] The device (10) may be configured to calculate a change value of a feature point on an acquired image based on the uniformity of the distribution of the feature points (S1604). At this time, it is necessary to calculate the change value of the same feature point in the image(s) acquired periodically. As the change value of the feature point, any one of the average value, median value, maximum value of the motion vector of the feature point, or the change in the number of feature points may be used.

[0172] The device (10) may be configured to obtain the update time of the homography matrix based on the change value of the calculated feature point (S1605).

[0173] Referring to FIG. 17, an example of determining the update time of a homography matrix is ​​illustrated. FIG. 17 (a) shows a case where the change value of a feature point increases over time. According to this, the magnitude of the accumulated change value over time gradually increases. As an example, the point in time when the magnitude of the accumulated change value exceeds a preset threshold can be determined as the update time of the homography matrix.

[0174] Figure 17(b) illustrates a case where the change value of a feature point remains constant over time. According to this, the magnitude of the accumulated change value increases in direct proportion to time. As an example, the point in time when the magnitude of the accumulated change value exceeds a preset threshold can be determined as the update point of the homography matrix. It is expected that the update point of the homography matrix in Figure 17(b) will be later than in Figure 17(a).

[0175] Figure 17 (c) shows a case where the change value of a feature point increases over time and then decreases again. According to this, since the change value of the feature point is almost constant over time, it can be configured to determine that an update of the homography matrix is ​​not necessary.

[0176]

[0177] Additionally, the device (10) may be configured to output a user interface component corresponding to a change value of a feature point to a human-machine interface (HMI).

[0178] Referring to FIG. 18, a user interface (1030) output to an HMI such as a display is illustrated. The user interface (1030) may be configured to include a user interface component (181) containing information about the feature point change amount (rotation angle and direction) and the homography matrix update time (i.e., the expected time for manual inspection), and a user interface component (182) containing information about the change in the magnitude of the change value over time.

[0179] Referring to FIG. 19, a user interface (1030) output to an HMI such as a display is illustrated. The user interface (1030) may be configured to include a user interface component (191) that includes information regarding feature point change amounts (rotation angle and direction), the time of homography matrix update (i.e., the expected time of manual inspection), and an automatic calibration completion notification.

[0180] Additionally, referring to FIG. 19, the reference image and the acquired image in which the change value of the accumulated feature points exceeds a preset threshold can be overlapped and output to the user interface (1030).

[0181] Referring to FIG. 20, a user interface (1030) output to an HMI such as a display is illustrated. The user interface (1030) may be configured to include user interface components (201, 202) for object regions that have changed over time. The user interface component (201) is a portion of a region in a reference image, and the user interface component (202) represents a portion of a region in an acquired image corresponding to the said region. Through the user interface component (203), the changes in the object on the reference image and the acquired image may be displayed.

[0182] Referring to FIG. 21, a user interface (1030) output to an HMI such as a display is illustrated. The user interface (1030) may be configured to include a user interface component that displays on a map the location where a feature point change occurred according to the degree of the feature point change amount (big, mid, small). Additionally, the user interface (1030) may be configured to include a user interface component that displays on a map the location where a feature point change (illustrated in FIG. 20) occurred.

[0183]

[0184] FIG. 22 illustrates a flowchart of a method for correcting the settings of a pre-installed camera based on GPS (global positioning system) tracking according to the present invention. The process of FIG. 22 is performed by a device (10) for correcting camera settings, and below, it will be described simply as being performed by the device (10).

[0185] The device (10) can be configured to detect an object in an acquired image (S2201).

[0186] The device (10) may be configured to obtain estimated GPS information of a detected object based on a homography matrix (S2202). The homography matrix represents the transformation relationship between two coordinate systems (planes), and the estimated GPS information of an object detected in an image may be obtained by using a homography matrix that defines the transformation relationship between image coordinates (planes) and GPS coordinates (planes).

[0187] The device (10) may be configured to calculate the difference between the acquired estimated GPS information and the actual GPS information of the detected object (S2203). The actual GPS information of the detected object may be included in the information collected from the object via V2X communication. For example, GPS information may be included as location information of the sender (i.e., the object) in a basic safety message (BSM) or a personal safety message (PSM).

[0188] Meanwhile, the acquired estimated GPS information can be updated over time, and even in such cases, the acquired estimated GPS information can be updated based on the movement speed information (included in the BSM or PSM) and time information received from the detected object. For example, it is expressed as P_estimate = P_prev + V*t, where P_prev is the acquired estimated GPS information and P_estimate represents the updated acquired estimated GPS information. V represents the movement speed of the object, and t represents the difference between the current time and the time information (e.g., timestamp information) included in a message such as the BSM or PSM.

[0189] The device (10) may be configured to determine whether the calculated difference exceeds a threshold (S2204).

[0190] The device (10) may be configured to update the homography matrix or perform calibration on the camera that captured the acquired image based on the calculated difference exceeding a threshold. Meanwhile, it may be configured to update the homography matrix or perform calibration on the camera that captured the acquired image only when the number of objects whose calculated difference exceeds the threshold exceeds a preset number.

[0191] Based on the fact that the calculated difference does not exceed a threshold, the device (10) can return to the object detection procedure.

[0192] Alternatively, the device (10) may be configured to detect multiple objects in an acquired image and to acquire the displacement amount of the acquired estimated GPS information and the actual GPS information of each object (e.g., distance and direction information between the two GPS information). If the displacement amount of the two GPS information for each of the multiple objects is within a threshold range, that is, if the displacement amount of the two GPS information for each of the multiple objects is similar, it may be determined that the camera that took the image has changed position or is misaligned. Accordingly, the device (10) may be configured to update the homography matrix or perform calibration on the camera that took the acquired image.

[0193]

[0194] FIG. 23 illustrates a block diagram of a device for setting correction of a camera according to the present invention.

[0195] A device (10) for correcting camera settings may be configured to correct the settings of an installed camera or update a homography matrix. Hereinafter, it will be simply referred to as “device (10)”.

[0196] The device (10) may include a memory (100) configured to store at least one feature point feature, image coordinates, actual GPS coordinates, actual converted GPS coordinates, and a homography matrix for a pre-installed camera.

[0197] The device (10) may include a processor (101) configured to acquire an image and, upon detection of a change in feature points in the acquired image, update a homography matrix stored in memory (100) or update the feature points.

[0198] The processor (101) may be configured to update the homography matrix or update the feature points based on the detection of a change in the feature points.

[0199] The processor (101) may be configured to acquire motion vectors of at least some of the feature points in the acquired image. The processor (101) may be configured to update a homography matrix based on the acquired motion vectors. Here, the motion vectors may be acquired based on the location of the feature points on the reference image and the location of the feature points on the acquired image.

[0200] The processor (101) may be configured to acquire motion vectors of feature points that do not overlap with the area (bbox) for the object detected in the acquired image.

[0201] The processor (101) may be configured to divide the area of ​​the acquired image into four sections based on the centroid of the feature points that do not overlap with the area (bbox) for the object detected in the acquired image, select a preset number of feature points in each of the four divided areas of the acquired image, and acquire the motion vector of the selected feature points.

[0202] The processor (101) may be configured to periodically acquire images and check whether the distribution of feature points in the acquired images is uniform. The processor (101) may be configured to calculate a change value of feature points in the periodically acquired images based on whether the distribution of feature points is uniform. The processor (101) may be configured to acquire an update time of the homography matrix based on the calculated change value. Here, the change value of the feature points may be configured to include at least one of the average value or median value of the motion vector of the feature points, the maximum value of the motion vector of the feature points, or a change in the number of the feature points.

[0203] Additionally, the processor (101) may be configured to output to a human-machine interface (HMI), such as a display, the time when an update of the homography matrix is ​​required or the time when a setting check of an installed camera is required.

[0204] The processor (101) can be configured to output a user interface component corresponding to a change value of a feature point to an HMI such as a display.

[0205] The processor (101) may be configured to output a user interface component corresponding to a change value of a feature point to an HMI such as a display, along with at least two images among the images acquired periodically.

[0206] Additionally, the processor (101) may be configured to obtain estimated GPS information of at least one detected object based on a homography matrix. The processor (101) may be configured to calculate the difference between the estimated GPS information and the actual GPS information of at least one detected object, and to determine whether to perform calibration of the installed camera or update of the homography matrix based on the calculated difference.

[0207] The processor (101) may be configured to update the homography matrix using the changed image coordinates of the feature points as the change in image coordinates of at least some of the feature points in the acquired image exceeds a threshold range.

[0208] The processor (101) may be configured to set a window for searching for feature points in an acquired image, detect features of a stored feature point in the set window, and acquire changed image coordinates of a feature point corresponding to the detected features.

[0209] More specifically, the homography matrix can be updated using the transformed image coordinates of the feature points and the actual GPS coordinates or actual transformed GPS coordinates of the feature points. Additionally, among the stored feature points, those for which no features were detected in the acquired image can be configured not to be used in updating the homography matrix.

[0210] The processor (101) can be configured to update features for stored feature points after the homography matrix is ​​updated, and to update the actual transformed GPS coordinates of the stored feature points using the updated homography matrix.

[0211] The processor (101) may be configured to determine whether to update the homography matrix or to determine that the updated homography matrix is ​​valid by using at least one of the number of feature points detected in the acquired image, the distance between a preset number of feature points in the acquired image, or the difference between the actual converted GPS coordinates of a preset number of feature points and the actual GPS coordinates according to the updated homography matrix and the changed image coordinates of the feature points.

[0212] The processor (101) may be configured to determine to update the homography matrix or to determine that the updated homography matrix is ​​valid if the number of feature points detected in the acquired image is greater than or equal to a preset number, the distance between the preset number of feature points in the acquired image is within a preset distance, and the difference between the actual converted GPS coordinates of the preset number of feature points acquired using the updated homography matrix and the changed image coordinates and the actual GPS coordinates is within a preset range.

[0213] Additionally, the processor (101) may be configured to update feature points as at least some of the feature points are not detected in the acquired image.

[0214] Accordingly, the processor (101) may be configured to acquire a new feature point in the acquired image through a corner detection algorithm, acquire actual transformed GPS coordinates of the new feature point using a homography matrix, and acquire features of the new feature point. The acquired new feature point may be located in an area outside a pre-specified exclusion area in the acquired image.

[0215] Additionally, the device (10) may further include a camera (102). The camera (102) refers to a camera that has already been installed.

[0216] Additionally, the device (10) may further include a display (103) configured to output an acquired image or an image with distortion corrected. Additionally, the display (103) may be configured to output a user interface for displaying changes in feature points based on motion vectors, such as those shown in FIGS. 18 to 21.

[0217]

[0218] According to the present invention, when the position or orientation of the camera changes from that of the initial installation, the image coordinates of the feature points and the homography matrix are updated through the above process, thereby ensuring the accuracy of the estimated GPS information corresponding to the image coordinates continuously. Furthermore, according to the present invention, when the environment in which the camera is capturing changes, new feature points are established and information regarding them is updated through the above process, thereby enabling continuous estimation of the object's location even with environmental changes.

[0219]

[0220] Meanwhile, operations, processes, etc. described with reference to FIG. 1 to FIG. 22, which are not described with reference to FIG. 23, may be performed by a processor (101) or a device (10) for correcting camera settings.

[0221]

[0222] In addition, as another aspect of the present invention, the operation of the above-described proposal or invention may be provided as code that can be implemented, practiced, or executed by a "computer" (a comprehensive concept including a system on chip (SoC) or (micro)processor, etc.), or as a computer-readable storage medium or computer program product that stores or contains said code, and the scope of the present invention may be extended to said code or as a computer-readable storage medium or computer program product that stores or contains said code.

[0223] Additionally, the functions of the elements disclosed herein include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, and may be implemented using circuits or processing circuits configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it contains transistors and other circuits within it. In this specification, circuitry, unit, or means means hardware that performs or is programmed to perform the described functions. Such hardware may be any hardware disclosed in this specification or hardware known to those skilled in the art, and may be programmed or configured to perform the described functions. Where the hardware is a processor, it may be considered a type of circuit, wherein said circuitry, means, or unit is a combination of hardware and software, said software is used to configure said hardware and / or processor.

[0224] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention as described in the following claims. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. As a device for calibrating the settings of an installed camera, A memory storing at least one feature point feature, image coordinates, actual GPS coordinates, actual transformed GPS coordinates, and a homography matrix; and A processor configured to acquire an image and update the homography matrix or update the feature points according to the detection of a change in the feature points in the acquired image. , device.

2. In paragraph 1, the processor At least some of the motion vectors of the feature points are obtained from the above-mentioned acquired image, and It is configured to update the homography matrix based on the acquired motion vector, and The above motion vector is obtained based on the position of the feature point on the reference image and the position of the feature point on the acquired image. , device.

3. In paragraph 2, the processor A method configured to acquire motion vectors of feature points that do not overlap with the area (bbox) for the object detected in the above-mentioned acquired image. , device.

4. In paragraph 3, the processor The area of ​​the acquired image is divided into four sections based on the centroid of feature points that do not overlap with the area for the object detected in the acquired image, and Select a preset number of feature points in each of the four divided regions of the above-mentioned acquired image, and Configured to acquire the motion vector of the selected feature point above , device.

5. In paragraph 1, the processor periodically acquire images, and Check whether the distribution of the feature points in the above-acquired image is uniform, and Calculate the change value of the feature points in the periodically acquired images based on the uniform distribution of the feature points, and Configured to obtain the update time of the homography matrix based on the above calculated change value , device.

6. In Paragraph 5, The change value of the above feature point includes at least one of the average value or median value of the motion vector of the above feature point, the maximum value of the motion vector of the above feature point, or the change in the number of the above feature points. , device.

7. In paragraph 6, the above processor Configured to output to a human-machine interface (HMI) the time when an update of the above homography matrix is ​​required or the time when a setting check of the above-mentioned installed camera is required. , device.

8. In paragraph 6, the above processor A user interface component corresponding to the change value of the above feature point is configured to output to a human-machine interface (HMI). , device.

9. In paragraph 6, the above processor A user interface component corresponding to the change value of the above feature point is configured to be output to a human-machine interface (HMI) together with at least two images among the periodically acquired images. , device.

10. In paragraph 1, the processor Estimated GPS information of at least one object detected based on the above homography matrix is ​​obtained, and Calculate the difference between the estimated GPS information and the actual GPS information of at least one detected object, and Configured to determine whether to perform calibration of the installed camera or update of the homography matrix based on the difference calculated above. , device.

11. In paragraph 1, the processor Configured to update the homography matrix using the changed image coordinates of the feature points as the change in image coordinates of at least some of the feature points in the acquired image exceeds a threshold range , device.

12. In paragraph 11, the above processor A window for searching feature points in the above-mentioned acquired image is set, and Detecting the features of the stored feature points in the window set above, and Configured to acquire changed image coordinates of feature points corresponding to the above-detected features , device.

13. In paragraph 11, the homography matrix is Updated using the changed image coordinates of the above feature point and the actual GPS coordinates or actual converted GPS coordinates of the above feature point , device.

14. In claim 11, among the stored feature points, feature points for which no feature is detected in the acquired image are configured not to be used in updating the homography matrix. , device.

15. In paragraph 11, the above processor Once the above homography matrix is ​​updated, Configured to update features for the stored feature points and update the actual transformed GPS coordinates of the stored feature points using the updated homography matrix , device.

16. In paragraph 11, the above processor The number of feature points detected in the above-mentioned acquired image, The distance between the feature points in the above-mentioned acquired image, or A method configured to determine whether to update the homography matrix or to determine that the updated homography matrix is ​​valid by using at least one of the difference between the actual transformed GPS coordinates of the feature point and the actual GPS coordinates according to the updated homography matrix and the changed image coordinates of the feature point. , device.

17. In paragraph 16, the above processor A device configured to determine to update the homography matrix or determine that the updated homography matrix is ​​valid if the number of feature points detected in the acquired image is greater than or equal to a preset number, the distance between the preset number of feature points in the acquired image is within a preset distance, and the difference between the actual converted GPS coordinates of the preset number of feature points acquired using the updated homography matrix and the changed image coordinates and the actual GPS coordinates is within a preset range.

18. In paragraph 1, the processor, Configured to update the feature points as at least a portion of the feature points are not detected in the acquired image. , device.

19. In paragraph 18, the above processor, New feature points are obtained from the above-mentioned acquired image through a corner detection algorithm, and It is configured to obtain actual transformed GPS coordinates of the new feature point using the above homography matrix and to obtain features of the new feature point, A device in which the newly acquired feature point is located in an area outside a pre-specified exclusion area in the acquired image.

20. A method for correcting the settings of an installed camera, wherein the method is performed by a device for correcting camera settings, and Step to acquire an image; The method includes the step of updating a previously stored homography matrix or updating the feature points based on the detection of a change in the feature points in the acquired image. , method.

Images