Traffic law violation monitoring system using camera and lidar
The camera-LiDAR fusion system addresses adverse weather challenges in traffic surveillance by integrating sensor data and advanced algorithms for precise violation detection, improving safety and efficiency.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHUNGBUK NAT UNIV IND ACADEMIC COOP FOUNDATION
- Filing Date
- 2025-08-14
- Publication Date
- 2026-06-04
AI Technical Summary
Existing camera-based traffic surveillance systems struggle with reduced accuracy and increased false detections during adverse weather conditions, such as rain, snow, and fog, leading to impaired traffic violation detection and safety.
A camera-LiDAR sensor fusion system that includes preprocessing, object detection, bad weather detection, object tracking, and traffic law violation determination units, utilizing sensor data synchronization, DBSCAN clustering, Kalman filtering, and SVM for robust performance in adverse weather.
Enhances traffic safety and efficiency by accurately detecting violations like accidents and illegal parking in real-time, even in adverse weather, supporting smart traffic management and autonomous vehicles, and aiding law enforcement.
Smart Images
Figure KR2025012405_04062026_PF_FP_ABST
Abstract
Description
Traffic violation monitoring system using cameras and LiDAR
[0001] The present invention relates to a traffic law violation monitoring system, and more specifically, to a traffic law violation monitoring system using a camera and a lidar to respond to adverse weather conditions.
[0002] Existing intelligent traffic surveillance systems primarily utilize cameras to monitor traffic conditions and detect traffic violations such as traffic accidents, jaywalking, and illegal parking. However, these camera-based traffic surveillance systems have limitations in accuracy due to restricted visibility and degraded image quality under adverse weather conditions, such as rain, snow, and fog.
[0003] Figure 1 illustrates the results of camera object detection in adverse weather conditions, and illustrates the results of camera recognition and LiDAR data in adverse weather conditions.
[0004] As shown in Figure 1, camera object detection performance deteriorates in adverse weather conditions, and false detections and non-detections increase. In such adverse weather conditions, the image processing performance of the camera deteriorates, making it difficult to accurately detect traffic violations, which threatens road safety and disrupts traffic flow.
[0005] LiDAR (light detection and ranging) is a sensor that uses lasers to acquire three-dimensional information about the surrounding environment, providing relatively stable detection performance in adverse weather conditions.
[0006] Figure 2 illustrates the results of LiDAR object detection in adverse weather conditions, showing the results of detecting moving objects in adverse weather conditions, such as vehicles.
[0007] As shown in Figure 2, it can be confirmed that noise is included in the lidar data. Due to this increase in data noise, the error increases when tracking objects using lidar detection results. As such, lidar can also show limitations due to signal attenuation or increased noise under extreme weather conditions, such as heavy rainfall or snowfall.
[0008] The present invention has been devised to solve the aforementioned problems and aims to provide a camera-LiDAR sensor fusion-based traffic law violation monitoring system capable of efficiently monitoring and preventing traffic law violations, such as traffic accidents, jaywalking, and illegal parking, that occur in adverse weather environments.
[0009] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives will be clearly understood by a person skilled in the art from the description below.
[0010] The present invention, for achieving the above purpose, relates to a traffic law violation monitoring system comprising: a sensor unit equipped with a camera and a lidar; a preprocessing unit for preprocessing sensor data output from the camera and lidar of the sensor unit; an object detection unit for detecting an object in the data preprocessed by the preprocessing unit; a bad weather detection unit for determining bad weather using object detection result data output from the object detection unit; an object tracking unit for tracking an object detected by the object detection unit, wherein the object is tracked using bad weather determination result data from the bad weather detection unit; a sudden event determination unit for determining whether a sudden event occurs using object tracking result data output from the object tracking unit; and a traffic law violation determination unit for determining whether a traffic law is violated regarding the sudden event determined by the sudden event determination unit.
[0011] The above preprocessing unit can collect sensor data from the camera and the lidar, and synchronize and calibrate the collected sensor data.
[0012] The object detection unit can detect objects using bounding boxes with respect to camera data preprocessed by the preprocessing unit, and can detect objects by designating a region of interest as a road area with respect to LiDAR data preprocessed by the preprocessing unit and applying DBSCAN (Density-based spatial clustering of applications with noise) clustering.
[0013] The above-mentioned bad weather detection unit can determine bad weather using the bounding box output from the object detection unit and the clustering result.
[0014] The object tracking unit above can use a Kalman filter to integrate object detection result data using the camera data and object detection result data using the LiDAR data, and perform object tracking on the integrated object data.
[0015] According to the present invention, by efficiently monitoring and preventing traffic law violations such as traffic accidents, jaywalking, and illegal parking, it is possible to improve road safety and contribute to smart traffic management. Furthermore, by maintaining stable performance even in adverse weather conditions, it facilitates smooth traffic flow and enhances public safety.
[0016] Furthermore, by utilizing the present invention, traffic management and control centers can detect traffic violations such as traffic accidents, jaywalking, and illegal parking in real time to optimize traffic flow and improve road safety. Additionally, as a core element of a smart city, it can contribute to effectively monitoring and managing traffic conditions throughout the city.
[0017] In addition, by utilizing the present invention, the safety and reliability of an autonomous vehicle can be enhanced through stable recognition of the surrounding environment even in adverse weather conditions in an autonomous vehicle sensor system, and accidents can be prevented by detecting and warning of pedestrians or obstacles in situations where the driver's field of vision is limited in a V2X environment.
[0018] Furthermore, according to the present invention, it can assist police and traffic enforcement agencies in efficiently monitoring and cracking down on violations such as illegal parking and jaywalking during traffic law enforcement. Additionally, it can assist in rapid response and alleviation of traffic congestion by providing real-time information during traffic disruptions caused by inclement weather.
[0019] With the recent increase in interest in traffic safety and efficiency, the demand for intelligent traffic monitoring systems is continuously rising. The sensor fusion technology provided by this invention, which offers high accuracy even in adverse weather conditions, can secure a competitive advantage over rivals. Furthermore, by reducing traffic accidents and enhancing road safety, it is possible to lower social costs and improve the quality of life for the public. Additionally, according to this invention, continuous revenue generation is possible through system sales, maintenance services, and data sales. In conclusion, this invention can be widely utilized in various fields such as traffic management, autonomous driving, and public safety, and possesses high commercialization potential based on its technical originality and marketability. Through this, it can improve traffic safety and efficiency and contribute to the establishment of a smart and safe traffic environment.
[0020] Figure 1 illustrates the results of camera object detection in adverse weather conditions.
[0021] Figure 2 illustrates the results of LiDAR object detection in adverse weather conditions.
[0022] FIG. 3 is a block diagram showing the configuration of a traffic law violation monitoring system according to one embodiment of the present invention.
[0023] FIG. 4 is a block diagram illustrating the configuration of a preprocessing unit according to one embodiment of the present invention.
[0024] FIG. 5 illustrates the synchronization and calibration results in a preprocessing unit according to one embodiment of the present invention.
[0025] FIG. 6 is a block diagram illustrating the configuration of a bad weather detection unit according to one embodiment of the present invention.
[0026] Figure 7 illustrates the frequency domain transformation result in an object detection bounding box.
[0027] FIG. 8 is a block diagram illustrating the configuration of an object tracking unit according to one embodiment of the present invention.
[0028] FIG. 9 is a diagram defining parameters used in a formula according to one embodiment of the present invention.
[0029] FIG. 10 illustrates a clustering detection result according to one embodiment of the present invention.
[0030] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.
[0031] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0032] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0033] Furthermore, in the description referring to the attached drawings, identical components are assigned the same reference numeral regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the present invention, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the present invention, such detailed description is omitted.
[0034] FIG. 3 is a block diagram showing the configuration of a traffic law violation monitoring system according to one embodiment of the present invention.
[0035] Referring to FIG. 3, a traffic law violation monitoring system according to one embodiment of the present invention includes a sensor unit (100), a preprocessing unit (200), an object detection unit (300), an adverse weather detection unit (400), an object tracking unit (500), a sudden occurrence determination unit (600), and a traffic law violation determination unit (700).
[0036] The sensor unit (100) is equipped with a camera and a lidar.
[0037] The preprocessing unit (200) performs the role of preprocessing sensor data output from the camera and lidar of the sensor unit (100).
[0038] The object detection unit (300) plays the role of detecting objects in the data preprocessed by the preprocessing unit (200).
[0039] The bad weather detection unit (400) determines bad weather using object detection result data output from the object detection unit (300).
[0040] The object tracking unit (500) tracks the object detected by the object detection unit (300), and tracks the object using the bad weather judgment result data from the bad weather detection unit (400).
[0041] The sudden occurrence determination unit (600) determines whether a sudden occurrence occurs using the object tracking result data output from the object tracking unit (500).
[0042] In the present invention, the sudden occurrence determination unit (600) can determine the type of sudden occurrence using object tracking result data.
[0043] In one embodiment of the present invention, the sudden occurrence determination unit (600) can determine 2D type object location information as coordinate system information, 3D type object location information as latitude and longitude information, object speed, object direction, blocked lane, and sudden occurrence type, and detect the section where the sudden occurrence situation occurred, location information, and whether the sensor unit is faulty.
[0044] The traffic law violation judgment unit (700) determines whether there is a traffic law violation regarding the sudden situation determined by the sudden situation judgment unit (600).
[0045] In the present invention, the traffic law violation judgment unit (700) can determine whether there is a traffic law violation according to the type of sudden event. For example, the traffic law violation judgment unit (700) can determine traffic law violations such as traffic accidents, driving against traffic, illegal U-turns, and illegal parking according to the type of sudden event.
[0046] The preprocessing unit (200) can preprocess data by collecting sensor data from a camera and a lidar, and synchronizing and calibrating the collected sensor data.
[0047] The object detection unit (300) can detect objects by using a neural network to detect bounding boxes for camera data preprocessed by the preprocessing unit (200), and can detect objects by designating a region of interest as a road area for the LiDAR data preprocessed by the preprocessing unit (200) and applying DBSCAN (Density-based spatial clustering of applications with noise) clustering.
[0048] The bad weather detection unit (400) can determine bad weather using the bounding box and clustering result output from the object detection unit (300).
[0049] The object tracking unit (500) can use a Kalman filter to integrate object detection result data using camera data and object detection result data using LiDAR data, and perform object tracking on the integrated object data.
[0050] In the present invention, the object tracking unit (500) can improve tracking performance by adjusting the measurement noise of the Kalman filter.
[0051]
[0052] FIG. 4 is a block diagram illustrating the configuration of a preprocessing unit according to one embodiment of the present invention.
[0053] Referring to FIG. 4, the preprocessing unit (200) comprises a sensor data collection unit (210), a sensor data synchronization unit (220), and a sensor data calibration unit (230).
[0054] The sensor data collection unit (210) collects sensor data from the camera and the lidar.
[0055] The sensor data synchronization unit (220) plays the role of synchronizing camera data and lidar data.
[0056] The sensor data calibration unit (230) performs calibration on the synchronized data.
[0057] FIG. 5 illustrates the synchronization and calibration results in a preprocessing unit according to one embodiment of the present invention.
[0058] In Fig. 5, images of the sensor data before and after synchronization and calibration can be compared.
[0059]
[0060] FIG. 6 is a block diagram illustrating the configuration of a bad weather detection unit according to one embodiment of the present invention.
[0061] Referring to FIG. 6, the bad weather detection unit (400) includes a data conversion unit (410), a bad weather determination unit (420), and a data reliability unit (430).
[0062] The data conversion unit (410) converts the bounding box data output from the object detection unit (300) into the frequency domain and converts the clustering result data into point cloud distribution data.
[0063] Figure 7 illustrates the frequency domain transformation result in an object detection bounding box. In Figure 7, the left side shows the domain transformation result image in normal conditions, and the right side shows the domain transformation result image in adverse weather conditions.
[0064] As shown in Figure 7, when converting the bounding box to the frequency domain, in the case of bad weather conditions, the noise increases and the pattern changes.
[0065] The bad weather determination unit (420) determines bad weather using frequency domain transformed bounding box data and determines bad weather using point cloud dispersion data and SVM (Support Vector Machine).
[0066] The data reliability unit (430) calculates a first reliability score using a camera reliability function for bounding box data and calculates a second reliability score using a LiDAR reliability function for point cloud dispersion data using a neural network.
[0067] The object tracking unit (500) can correct the measurement noise of the Kalman filter using the first reliability score and the second reliability score calculated by the data reliability unit. This can improve the performance of object tracking.
[0068] FIG. 8 is a block diagram illustrating the configuration of an object tracking unit according to one embodiment of the present invention.
[0069] Referring to FIG. 8, the object tracking unit (500) includes a camera object tracking filter (510), a LiDAR object tracking filter (520), and an integrated Kalman filter (530).
[0070] The camera object tracking filter (510) filters object tracking using camera data.
[0071] The LiDAR object tracking filter (520) filters object tracking using LiDAR data.
[0072] As such, the present invention filters object tracking for each sensor of the camera and lidar respectively.
[0073] The integrated Kalman filter (530) performs object tracking only on the matching results between the object detection results through the camera object tracking filter (510) and the object detection results through the LiDAR object tracking filter (520).
[0074] In the present invention, object tracking for each sensor of the camera and lidar can be based on SORT (Simple Online and Realtime Tracking), an image-based object tracking algorithm.
[0075] FIG. 9 is a table defining parameters used in a formula according to an embodiment of the present invention, defining parameters used in a formula required for a Kalman filter.
[0076] Referring to FIG. 9, X is the state vector of the Kalman filter (the state variable value to be estimated), Z is the measurement vector of the Kalman filter (the data value measured from the sensor), μ and ν are the center coordinate values in the image plane of the detected object (pixel coordinate system), s is the scale of the bounding box, and r is the aspect ratio of the bounding box. is the center coordinate (speed concept), ε is the rate of change in scale (a concept of speed), F is the state transition matrix (a concept that predicts the next state in a Kalman filter), H is the measurement transition matrix (expresses the relationship between the measurement vector and the state vector in a Kalman filter), and R i is the measurement noise per sensor, R sensor is the reliability function value per sensor, R Nor is the normalized value per sensor, R total is the measurement noise of the integrated Kalman filter.
[0077] The state vector X is equal to the following equation.
[0078] [Mathematical Formula 1]
[0079]
[0080] The state transition matrix F is equal to the following equation.
[0081] [Mathematical Formula 2]
[0082]
[0083]
[0084] The measurement vector Z is equal to the following equation.
[0085] [Mathematical Formula 3]
[0086]
[0087]
[0088] The measurement transition matrix H is given by the following equation.
[0089] [Mathematical Formula 4]
[0090]
[0091] In the present invention, the update of the Kalman filter is the same as that of a general Kalman filter.
[0092] Next, the LiDAR object tracking algorithm is as follows.
[0093] The state vector X is equal to the following equation.
[0094] [Mathematical Formula 5]
[0095]
[0096]
[0097] The state transition matrix F follows the constant velocity model and is given by the following equation.
[0098] [Mathematical Formula 6]
[0099]
[0100]
[0101] The measurement vector Z is equal to the following equation.
[0102] [Mathematical Formula 7]
[0103]
[0104]
[0105] The measurement transition matrix H is given by the following equation.
[0106] [Mathematical Formula 8]
[0107]
[0108]
[0109] After object tracking using LiDAR data, the estimated state When saying, the estimated state μ and ν are calculated by projecting the x and y values onto the image coordinate system.
[0110] In the case of conventional two-camera object tracking, the state vector, measurement transition matrix, and measurement noise are as shown in Equation 9 below.
[0111] [Mathematical Formula 9]
[0112]
[0113] In the present invention, as an object fusion technology for cameras and lidar, the equations for Z, H, and R in Equation 9 are modified, and are as follows in Equation 10.
[0114] [Mathematical Formula 10]
[0115]
[0116]
[0117] In this case, the state vector is the same as before, and the measurement noise R for the camera sensor C is determined by the camera reliability function, and the measurement noise R for the LiDAR sensor L It is determined by the LiDAR reliability function.
[0118] The camera reliability function calculates the error for the reconstructed result using Mean Squared Error (MSE), and uses the output MSE to R C Determines.
[0119] [Mathematical Formula 11]
[0120]
[0121] [Mathematical Formula 12]
[0122]
[0123] Here, α is a hyperparameter determined through experiments.
[0124] The LiDAR confidence function consists of the variance of objects detected through clustering.
[0125]
[0126] FIG. 10 illustrates a clustering detection result according to one embodiment of the present invention.
[0127] In the case of LiDAR sensors, if data is determined to be abnormal using SVM, the confidence function is determined by utilizing the variance value of the clustering detection result, which is a set of point clouds.
[0128] [Mathematical Formula 13]
[0129]
[0130] [Mathematical Formula 14]
[0131]
[0132]
[0133] In one embodiment of the present invention, camera measurement noise and lidar measurement noise are normalized so as not to diverge, and additionally corrected with β and γ so that the values are not too large.
[0134] [Mathematical Formula 15]
[0135]
[0136] Mathematical formula 15 is used to properly adjust the scale.
[0137] As such, the present invention improves the performance of object tracking by utilizing measurement noise, and can detect sudden occurrence types using this.
[0138]
[0139] Although the present invention has been described above using several preferred embodiments, these embodiments are illustrative and not limiting. Those skilled in the art will understand that various changes and modifications can be made without departing from the spirit of the invention and the scope of rights set forth in the appended claims.
Claims
1. A sensor unit equipped with a camera and a lidar; A preprocessing unit for preprocessing sensor data output from the camera and the lidar of the sensor unit; An object detection unit for detecting objects in data preprocessed in the above preprocessing unit; A bad weather detection unit for determining bad weather using object detection result data output from the object detection unit above; An object tracking unit for tracking an object detected by the object detection unit, wherein the object is tracked using adverse weather judgment result data from the adverse weather detection unit; An incident determination unit for determining whether an incident occurs using object tracking result data output from the object tracking unit; and A traffic law violation judgment unit that determines whether a traffic law has been violated regarding the sudden situation determined by the above-mentioned sudden situation judgment unit. A traffic violation monitoring system including 2. In Claim 1, A traffic law violation monitoring system characterized by the above-mentioned preprocessing unit collecting sensor data from the camera and the lidar, and synchronizing and calibrating the collected sensor data.
3. In Claim 2, A traffic law violation monitoring system characterized by the object detection unit detecting objects using bounding boxes with respect to camera data preprocessed by the preprocessing unit, designating a region of interest as a road area with respect to LiDAR data preprocessed by the preprocessing unit, and applying DBSCAN (Density-based spatial clustering of applications with noise) clustering.
4. In Claim 3, A traffic law violation monitoring system characterized by the above-mentioned bad weather detection unit determining bad weather using the bounding box output from the object detection unit and the clustering result.
5. In Claim 4, A traffic law violation monitoring system characterized by the object tracking unit integrating object detection result data using camera data and object detection result data using LiDAR data using a Kalman filter, and performing object tracking on the integrated object data.