Construction equipment safety system, and method for improving object detection accuracy by fusing multiple pieces of sensor data included in construction equipment safety system

Abstract

The present invention relates to a construction equipment safety system, and a method for improving object detection accuracy by fusing multiple pieces of sensor data included in the construction equipment safety system. The method according to an embodiment of the present invention comprises: a data collection step of collecting sensor data related to object detection from a sensor unit including a camera and a radar; and a coordinate fusion step of fusing coordinates for the sensor data of the camera and the radar included in the sensor unit, wherein in the coordinate fusion step, calibration is performed to align position coordinate data for the sensor data collected in the data collection step on the basis of the installation positions of the camera and the radar.

Description

A construction equipment safety system and a method for improving object detection accuracy by fusing multiple sensor data included in the construction equipment safety system.

[0001] The present invention relates to a construction equipment safety system and a method for improving object detection accuracy by fusing a plurality of sensor data included in the construction equipment safety system. According to the present invention, by fusing the coordinates of data detected by a plurality of radars and a plurality of cameras installed on the construction equipment, not only can the accuracy of object recognition be increased, but object recognition and obstacle identification specialized for the construction equipment safety system can also be achieved.

[0002] Various sensors are used in many construction equipment units operated at construction sites to ensure worker safety.

[0003] Commonly used sensors include cameras capable of recognizing the surrounding environment through video. Recently, with the integration of artificial intelligence, cameras are increasingly applying technology that distinguishes whether an approaching object is a person or an ordinary object.

[0004] In addition to cameras that collect images, radar sensors are also widely used. As is well known, a radar sensor is a sensor that emits electromagnetic waves and analyzes reflected signals to measure the distance, speed, and direction of an object.

[0005] At this time, radio wave-based sensors such as radar are of great help in ensuring safety at construction sites, as the distance to an object must be accurately measured; however, in the case of radar, it is not possible to know identification information of the object other than the distance to the object and its location coordinates. For example, it is not possible to know what type of object is detected (whether it is a person or a geographical feature, etc.).

[0006] On the other hand, as mentioned above, cameras equipped with AI technology capable of object recognition can distinguish the types of objects, but location data such as the distance to the object is detected less accurately than radar.

[0007] Therefore, when radar and cameras are installed and used together on construction equipment, they complement each other to detect objects more accurately. In this case, technology that fuses and processes data from each sensor is essential, and this is also called sensor fusion.

[0008] Meanwhile, as illustrated in FIG. 1, in the case of construction equipment (10), the camera (30) is installed at a relatively high position to have a downward-looking view, and the radar (20) has a forward-looking view at a lower position than the camera, taking into account the reflection component. This difference in viewing angle is a factor that makes sensor fusion in construction equipment more difficult than in automobiles, where the two sensors are generally installed to have a view in the same direction.

[0009] Furthermore, while the primary objective of safety systems applied to automobiles is to prevent collisions with objects while driving, making it important to accurately detect all obstacles regardless of distance or direction, the primary objective of safety systems applied to construction equipment is to prevent collisions with objects during operation, making it more important to accurately and quickly detect objects within the movement range of the work device.

[0010] One embodiment of the present invention aims to provide a system and method capable of accurately detecting the position and distance of an object by fusing sensor data in which a plurality of radars and a plurality of cameras detect an object.

[0011] In addition, one embodiment of the present invention aims to provide a system and method that enables construction equipment to quickly and accurately detect hazardous objects by intensively detecting the area with the highest risk of collision with an object during operation.

[0012] However, the problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned problems may be clearly understood based on the description below.

[0013] A method according to one embodiment of the present invention is a method for improving the detection accuracy of an object by fusing a plurality of sensor data, performed by a construction equipment safety system, and may include: a data collection step of collecting sensor data for detecting an object from a sensor unit including a camera and a radar; and a coordinate fusion step of fusing the coordinates of the sensor data of the camera and the radar included in the sensor unit.

[0014] Alternatively, the coordinate fusion step may perform a correction to align position coordinate data based on the installation positions of the camera and the radar with respect to the sensor data collected in the data collection step.

[0015] Alternatively, the coordinate fusion step may include: a radar coordinate generation step that generates three-dimensional radar coordinate data for the position of an object using distance data and azimuth data to the object measured by the radar; a camera coordinate conversion step that converts two-dimensional pixel coordinate data for the position of the object measured by the camera into three-dimensional camera coordinate data; and a coordinate correction step that performs a correction to align the radar coordinate data and the camera coordinate data by reflecting the relative positional difference between the radar and the camera installed on the construction equipment.

[0016] Alternatively, the sensor unit may include a plurality of radars installed at different locations.

[0017] Alternatively, the radar coordinate generation step may include: a step of generating radar local coordinate data for the position of an object for each radar using distance data and azimuth data to an object measured by each of the plurality of radars; and a step of converting each of the local coordinate data into radar world coordinate data by reflecting the installation position and installation angle of each radar, and then determining that objects located at the same coordinate value are considered as a single object.

[0018] Alternatively, the sensor unit may include a plurality of cameras installed at different locations.

[0019] Alternatively, the camera coordinate transformation step may include: a step of transforming the 2D pixel coordinate data of each of the plurality of cameras into camera normalized coordinate data based on the intrinsic parameters of each camera; a step of transforming each coordinate value of the camera normalized coordinate data into camera local coordinate data by reflecting the distance data measured by the radar; and a step of transforming each of the camera local coordinate data into camera world coordinate data by reflecting the installation position and installation angle of each camera, and then determining that objects located at the same coordinate value are considered as a single object.

[0020] Alternatively, the coordinate correction step may align the coordinates based on the camera by performing a transformation that reflects the positional and angular differences of the camera and the radar installed on the construction equipment in the three-dimensional radar coordinate data.

[0021] Alternatively, the data collection step may remove error components of the data based on the distance to the object measured by the radar and the intensity of the reflected wave from the object.

[0022] Alternatively, the coordinate fusion step may set a Region of Interest (ROI) based on the distance to an object measured by the radar only if the distance is within a preset threshold distance, and remove radar data outside the ROI from the target of the coordinate fusion.

[0023] Alternatively, the area of ​​interest may be set to a certain area around the work device based on the location of the work device of the construction equipment.

[0024] A construction equipment safety system according to one embodiment of the present invention may include: a sensor unit installed on construction equipment and comprising a camera and a radar; and a safety controller that receives sensor data detecting an object from the camera and the radar and fuses the coordinates of the sensor data of the camera and the radar.

[0025] Alternatively, the safety controller may perform a correction to align position coordinate data with respect to the sensor data based on the installation locations of the camera and the radar.

[0026] Alternatively, the safety controller may set a Region of Interest (ROI) based on the distance to an object measured by the radar only if the distance is within a preset threshold distance, and remove radar data outside the ROI from the target of coordinate fusion.

[0027] Alternatively, the area of ​​interest may be set to a certain area around the work device based on the location of the work device of the construction equipment.

[0028] According to at least one embodiment of the present invention, sensor data in which a plurality of radars and a plurality of cameras detect an object are fused together, and the measured distance of the radar is reflected in the depth information of the camera and the installation position and angle of each sensor is reflected to align the coordinates, thereby making it possible to accurately detect the position and distance of an object.

[0029] In addition, according to at least one embodiment of the present invention, by setting the area with the highest risk of collision with an object during operation as a region of interest based on the distance and / or coordinates detected by the radar, and excluding sensor data of objects outside the said region of interest from the data fusion target between sensors, it becomes possible for construction equipment to detect hazardous objects quickly and accurately.

[0030] Figure 1 shows the field of view of each camera and radar installed on construction equipment.

[0031] FIG. 2 is a block diagram of components that perform the function of a construction equipment safety system according to one embodiment of the present invention.

[0032] FIG. 3 is a diagram illustrating an example in which the coordinates of a plurality of sensors match by sensor data fusion of the present invention.

[0033] FIG. 4 is a flowchart showing the flow of a method according to one embodiment of the present invention.

[0034] Figure 5 is a flowchart showing the detailed flow of the coordinate fusion step of the flowchart of Figure 4 in more detail.

[0035] Figure 6 is a diagram showing overlapping detection areas between multiple radars.

[0036] Figure 7 shows an example of determining a single object for the overlapping detection area of ​​each camera and the overlapping detected object.

[0037] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.

[0038] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.

[0039] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.

[0040] The term “at least one of A or B” as used in the present disclosure should be interpreted as referring to A, B, and combinations of A and B.

[0041] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.

[0042] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0043] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”

[0044] The term “the N (N is a natural number)” used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but must be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.

[0045] The term “connection” as used in the present disclosure should be interpreted to include not only cases where the components are “directly connected,” but also cases where other components are “present” in between, and cases where they are “electrically connected” with other components in between.

[0046] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0047] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.

[0048] An embodiment of the present invention will be described in detail below with reference to the attached drawings.

[0049] FIG. 2 is a block diagram of components that perform the function of a construction equipment safety system according to one embodiment of the present invention.

[0050] Referring to FIG. 2, a construction equipment safety system (100) according to one embodiment of the present invention may include a sensor unit (110) and a safety controller (120). At this time, the construction equipment safety system (100) may be installed on the construction equipment (10) of FIG. 1.

[0051] In the embodiment of the present invention, the construction equipment (10) is described using an excavator as an example, but is not limited thereto. In the case of the excavator used as an example, the work device mentioned in this specification may include a bucket, an arm, a boom, etc.

[0052] The sensor unit (110) can detect objects around the construction equipment (10). The sensor unit (110) may include a camera unit (111) comprising one or more cameras (111a to 111c) and a radar unit (112) comprising one or more radars (112a to 112c).

[0053] In FIG. 2, the camera unit (111) and the radar unit (112) are each shown as including three cameras and three radars, but this is merely an example, and more than three cameras and radars may be installed, or only one may be installed. Also, the number of cameras and radars installed may not match each other.

[0054] Each camera and radar is installed in a position that allows it to view the outside of the construction equipment (10). For example, the first camera (111a) and the first radar (112a) may be installed on the left side of the construction equipment (10). For example, the second camera (111b) and the second radar (112b) may be installed on the right side of the construction equipment (10). For example, the third camera (111c) and the third radar (112c) may be installed on the rear of the construction equipment (10).

[0055] In the above example, it was stated that sensors are installed only in directions (left, right, rear) out of the driver's line of sight of the construction equipment (10), but sensors can also be additionally installed in front of the construction equipment (10).

[0056] Meanwhile, what is referred to as "camera" below may be understood to mean at least one camera included in the camera unit (111), and what is referred to as "radar" may be understood to mean at least one radar included in the radar unit (112).

[0057] The safety controller (120) is a component that controls the overall operation of the construction equipment safety system (100).

[0058] Here, the safety controller (120) may include all types of devices capable of processing data, such as a processor. Here, 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by code or instructions included in a program, for example. Examples of such data processing devices embedded in hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a neural processing unit (Neural Processing Unit), etc., but the scope of the present invention is not limited thereto.

[0059] The safety controller (120) can receive sensor data detecting an object from the sensors (camera and radar) included in the sensor unit (110). At this time, the transmission of sensor data can be done through the CAN (Controller Area Network) communication protocol.

[0060] Communication between each sensor included in the safety controller (120) and the sensor unit (110) may be performed directly, but may also be performed through the main controller (not shown) of the construction equipment (10). Here, the main controller refers to a controller that controls the driving motion, work motion, etc. of the construction equipment (10).

[0061] The safety controller (120) can fuse the coordinates of the sensor data of the camera and radar. Further details regarding the fusion of the camera and radar coordinates will be described later.

[0062] Additionally, the safety controller (120) can determine whether there is a dangerous situation after fusion of the coordinates of the camera and the radar. Here, a dangerous situation may refer, for example, to a situation where there is a high risk of collision between an object detected by the sensor and the construction equipment (10). The dangerous situation may be determined based on the distance between the work device and the object, the turning direction and speed of the construction equipment (10), the type of operation of the work device, etc.

[0063] A construction equipment safety system (100) according to one embodiment of the present invention may further include a display device (130).

[0064] The display device (130) may display visual information such as an around-view image generated using one or more cameras, information about an object whose coordinates are fused by the safety controller (120), and recognition information of an object determined by artificial intelligence. In a possible embodiment, the display device (130) may further display a warning message regarding a dangerous situation determined by the safety controller (120). In a possible embodiment, the display device (130) may be configured as a touchscreen.

[0065] A construction equipment safety system (100) according to one embodiment of the present invention may further include a communication module (140).

[0066] The communication module (140) is a configuration provided to communicate with the learning server (150) to be described later. It may be provided separately from the safety controller (120), but may also be included inside the safety controller (120) to form a single device. The communication module (140) may utilize a wireless communication method such as LTE, Wi-Fi, or BLE, or a wired communication method such as Ethernet.

[0067] A construction equipment safety system (100) according to one embodiment of the present invention may further include a learning server (150).

[0068] In a possible embodiment, the learning server (150) may communicate with the safety controller (120) to recognize and classify objects in the image using the camera sensor data (image) received from the safety controller (120), and transmit the results to the safety controller (120). In another possible embodiment, the learning server (150) may communicate with the safety controller (120) to download and update the learning program stored in the learning server (150) to the safety controller (120). In this case, object recognition may be performed by the safety controller (120) itself.

[0069] Meanwhile, the YOLO (You Only Look Once) model, which is widely known for providing excellent performance in analyzing multiple objects, may be used for the object recognition mentioned herein. Since the specific algorithm of the YOLO model is a known technology and is unrelated to the technical concept of the present invention, a detailed explanation is omitted.

[0070] Hereinafter, a method according to an embodiment of the present invention will be described. Specifically, the present invention is a method for fusing a plurality of sensor data to improve the object detection accuracy of construction equipment, and can be performed by the construction equipment safety system (100) described above. More specifically, the present invention can be performed by a safety controller (120) of the construction equipment safety system (100).

[0071] FIG. 3 is a diagram illustrating an example in which the coordinates of a plurality of sensors match by sensor data fusion of the present invention.

[0072] Figure 3(a) illustrates a situation where the coordinate data detected by the camera and the radar regarding an object do not match. This situation occurs due to differences in the installation positions of each sensor, differences in installation angles, and the use of different coordinate systems.

[0073] Meanwhile, in such cases, the distance to the object (person) detected by the radar cannot be applied to the person being viewed by the camera, so the exact distance from the construction equipment to the person cannot be known, and there is a problem that the detection accuracy is not improved even when multiple sensors are applied.

[0074] FIG. 3(b) illustrates a situation in which coordinate data of a person detected by a camera and a radar is matched by applying the method of the present invention, which will be described below. By matching the coordinate data of each sensor through the fusion of sensor data, the information about the object from the camera and the radar can be combined to accurately determine the type of object, the distance to the object, and the location of the object, thereby improving the accuracy of object detection and reducing the risk of safety accidents during the operation of construction equipment.

[0075] FIG. 4 is a flowchart showing the flow of a method according to one embodiment of the present invention, and FIG. 5 is a flowchart showing the detailed flow of the coordinate fusion step of FIG. 4 in more detail.

[0076] First, referring to FIG. 4, a method according to one embodiment of the present invention may include a data collection step (S100) and a coordinate fusion step (S200).

[0077] The data collection step (S100) is a step of collecting sensor data that detects an object from a sensor unit (110) including a camera and a radar.

[0078] As previously described, the sensor unit (110) can detect objects around the construction equipment (10). The sensor unit (110) may include a camera unit (111) comprising one or more cameras (111a to 111c) and a radar unit (112) comprising one or more radars (112a to 112c).

[0079] At this stage, a process to remove error components from the data can be carried out based on the distance to the object measured by the radar and the intensity of the reflected waves from the object.

[0080] Specifically, radar detects objects by transmitting radio waves and receiving signals reflected from them. However, in addition to the objects that are actually to be detected, there are error components that are reflected back from the ground or other sources. For example, when radio waves are reflected from the ground, there are cases where a signal is detected as if an object exists at a location where it does not, due to waves that bounce off in an unexpected direction. These error components are called clutter, and the accuracy of object detection can only be improved if these clutter components are removed first, thereby preventing the problem of error components being included in the subsequent fusion of sensor data.

[0081] As a specific embodiment of clutter removal, clutter removal can be performed based on information regarding the distance to an object and the intensity of reflected waves from the object at that distance, collected during a prior test process. That is, if the intensity of the reflected wave is too weak or too strong relative to the measured distance to the object, the sensor data detecting the object can be determined as a clutter component and removed.

[0082] As such, by performing clutter removal first and then proceeding with the coordinate fusion step (S200), the efficiency, speed, and accuracy of the computation are further improved.

[0083] The coordinate fusion step (S200) is a step of fusing coordinates for sensor data of a camera and a radar included in the sensor unit (110). In this step, correction can be performed to align position coordinate data based on the installation locations of the camera and the radar for the collected sensor data.

[0084] More specifically, referring further to FIG. 5, the coordinate fusion step (S200) may include a radar coordinate generation step (S210), a camera coordinate transformation step (S220), and a coordinate correction step (S240).

[0085] The radar coordinate generation step (S210) is a step of generating three-dimensional radar coordinate data for the position of an object using distance data and azimuth data to the object measured by the radar.

[0086] The radar coordinate generation step (S210) can generate radar local coordinate data for the position of the object for each radar using distance data and azimuth data to the object measured by each of the plurality of radars installed at different locations of the construction equipment (10). (S211)

[0087] Here, the distance data and azimuth data to the object measured by the radar refer to refined data from which clutter components have been removed in the coordinate collection step (S100). Since the distance to the object and the angle of view of the object are determined through the distance data and azimuth data measured by the radar, three-dimensional radar local coordinate data for the position of the object detected by the radar is generated.

[0088] When local coordinate data is generated for each radar, the local coordinate data is converted into radar world coordinate data by reflecting the installation position and installation angle of each radar, and objects located at the same coordinate value can be determined as a single object. (S212)

[0089] Figure 6 is a diagram showing overlapping detection areas between multiple radars.

[0090] Referring to Fig. 6, it can be seen that the first radar and the second radar have different detection areas due to having different installation positions and installation angles, but there is a certain overlapping detection area between the two.

[0091] Since the installation location of each radar varies, the local coordinate data for each radar is based on that specific radar as the origin. Because multiple radars have different reference points, duplicate objects will appear at different local coordinate locations. Therefore, to determine that a single object has been detected when duplicates exist, it is necessary to convert the local coordinate data of each radar into radar world coordinate data based on a single radar, utilizing the installation location and angle of each radar.

[0092] For example, when the first radar is the reference, the second radar and the third radar can be converted into world coordinate data through coordinate transformation using a position translation transformation matrix and a rotation translation transformation matrix that reflect the installation position and installation angle of the first radar.

[0093] After this conversion, duplicate detected objects with matching radar coordinates are determined to be the same object, thereby further improving detection accuracy.

[0094] Next, the camera coordinate conversion step (S220) is a step of converting 2D pixel coordinate data for the position of an object measured by the camera into 3D camera coordinate data.

[0095] The camera coordinate transformation step (S220) can transform the 2D pixel coordinate data of each of the plurality of cameras into camera normalized coordinate data based on the intrinsic parameters of each camera. (S221)

[0096] At this time, the intrinsic parameters of the camera are represented by the camera matrix K, and specifically, can be expressed as matrix 1 below.

[0097] [Matrix 1]

[0098]

[0099] Here, and ε is the focal length of the camera lens. Focal length is the distance between the lens and the image sensor, and it affects the field of view (FOV) and depth. Focal length indicates how much the image is magnified or reduced in the x-axis and y-axis directions when converting between 3D and 2D coordinates. and represents the x and y coordinates of the image sensor as the principal point.

[0100] For example, 2D pixel coordinates ( , If so, the following mathematical formula 1 can be applied to convert this into camera normalized coordinate data.

[0101] [Mathematical Formula 1]

[0102]

[0103] Here, is the inverse of the camera matrix.

[0104] However, the coordinate data transformed in this way is normalized coordinate data, and the depth information z is effectively 1, so it is not actually accurate 3D position data. Therefore, the camera's depth information must be supplemented to expand and transform it into camera local coordinate data.

[0105] The camera coordinate transformation step (S220) can convert each coordinate value of the camera normalized coordinate data into camera local coordinate data by reflecting the distance data measured by the radar. (S222)

[0106] At this time, the radar measurement distance data to be reflected in each normalized coordinate data can be selected as follows.

[0107] In the radar coordinate generation step (S210), radar world coordinate data is generated for multiple radars. Each coordinate value of this radar world coordinate data can be converted into camera 3D coordinate data. At this time, the installation position and installation angle of each camera can be reflected, and through coordinate transformation using a position translation transformation matrix and a rotation translation transformation matrix, the radar world coordinate data and the reference coordinates of the corresponding camera (e.g., the first camera) are aligned.

[0108] Afterwards, the radar world coordinate data converted into camera 3D coordinate data can be converted into 2D pixel coordinate data using a camera matrix (K) representing the intrinsic parameters of the corresponding camera (first camera).

[0109] Finally, among the pixel coordinate data of an object detected by the camera (first camera), data matching the coordinate value of the radar world coordinate data converted into 2D pixel coordinate data can be found. Once matching camera pixel coordinate data is found, the radar that calculated the matched radar world coordinate data can apply the measured distance to the normalized coordinate data converted into this pixel coordinate data to supplement the camera's depth information.

[0110] The radar's measurement distance is applied to the camera normalized coordinate data through the following mathematical formula 2 to produce the final camera local coordinate data (X,Y,Z).

[0111] [Mathematical Formula 2]

[0112]

[0113]

[0114]

[0115] Here, x, y, and z are the respective coordinate values ​​of the camera normalized coordinate data, and d is the measured distance of the radar.

[0116] Next, the camera coordinate transformation step (S220) transforms each camera local coordinate data into camera world coordinate data by reflecting the installation position and installation angle of each camera, and then determines that objects located at the same coordinate value are considered as a single object. (S223)

[0117] Since the installation location of each camera varies, the local coordinate data of each camera is based on that camera as the origin. Just like multiple radars, multiple cameras also have different reference points, so duplicate objects appear at different local coordinate locations. Therefore, to determine that a single object has been detected when duplicates are present, the local coordinate data of each camera is converted into camera world coordinate data based on a single camera, using the installation location and angle of each camera.

[0118] For example, when the first camera is the reference, the second and third cameras can be converted into world coordinate data through coordinate transformation using a position translation transformation matrix and a rotation translation transformation matrix that reflect the installation position and installation angle with respect to the first camera.

[0119] After this conversion, duplicate detected objects with matching camera coordinates are determined to be the same object, which can further improve detection accuracy.

[0120] Figure 7 shows an example of determining a single object for the overlapping detection area of ​​each camera and the overlapping detected object.

[0121] In FIG. 7(a), an example of the installation of four cameras is shown, and overlapping detection areas appear at the edges of the field of view of each camera. At this time, if the process of removing overlapping objects as described above is performed, it is possible to accurately determine that an object (or person) that is actually the same object (or person) appears with two coordinates as in FIG. 7(b) is a single object.

[0122] The coordinate correction step (S240), which is the final step of the coordinate fusion step (S200), is a step of performing correction to align the radar coordinate data and the camera coordinate data by reflecting the relative positional difference between the radar and the camera installed on the construction equipment (10).

[0123] Here, radar coordinate data and camera coordinate data refer to the radar world coordinate data and camera world coordinate data generated and transformed in steps S212 and S223, respectively.

[0124] In this step (S240), a transformation is performed to align the data of the camera and radar into the same three-dimensional coordinate system. A transformation is performed on the radar coordinate data to reflect the positional and angular differences in which the camera and radar are installed on the construction equipment (10), thereby aligning the coordinates based on the camera. At this time, a positional translation transformation matrix and a rotational translation transformation matrix may be used in the transformation.

[0125] For example, to convert to a coordinate system based on the camera view (reference), a conversion formula such as Equation 3 below can be used.

[0126] [Mathematical Formula 3]

[0127]

[0128] Here, is a matrix representing the position translation transformation from radar to camera, and is a rotational translation matrix that aligns coordinates by reflecting rotation based on angle difference.

[0129] In this case, one example of a transformation matrix can be as shown in matrix 2 and matrix 3 below. (Here, is the distance traveled from the radar origin to the camera origin, ~ represents the rotational component (direction vector) of each rotation axis.

[0130] [Matrix 2]

[0131]

[0132] [Matrix 3]

[0133]

[0134] Through this process, the same object recognized by the camera and radar can be accurately aligned in three-dimensional space. Meanwhile, since the camera and radar are installed at different heights, a correction that takes into account the difference in altitude may be added to correct the vertical position.

[0135] Meanwhile, the coordinate fusion step (S200) may further perform the step of setting a Region of Interest (ROI) based on the distance to an object measured by the radar only when the distance is within a preset threshold distance, and removing radar data that is outside the region of interest from the target of coordinate fusion (S230).

[0136] Construction equipment (10) travels at a low speed unlike ordinary automobiles due to its characteristics, so safety management regarding collisions with objects while working while stationary is a more important issue than collisions with objects while driving. In addition, during the turning process while working, the bucket, boom, and arm, which are the working devices, are the parts with the highest risk of collision with obstacles.

[0137] Therefore, unlike conventional automobiles, accurate information is not required for the detection data of all objects; it is more important to quickly detect accurate information regarding objects with a high risk of collision.

[0138] For example, considering the lengthwise size of the work device and the movement of the construction equipment during operation, a distance of approximately 10m from the sensor can be set as the collision risk range. At this time, the aforementioned threshold distance can be set to 10m. If the distance to the object measured by the radar exceeds 10m, there is not a great need for coordinate data fusion between camera data and radar data, and such radar data can be removed from the coordinate fusion target to increase the computational speed and efficiency of the safety controller (120).

[0139] Furthermore, when a detected object is displayed on the display device (130), the driver's attention can be focused on the object with a high risk of collision by processing it so that a bounding box is displayed only on the fused object.

[0140] As an additional embodiment, the aforementioned region of interest (ROI) may be set to a certain area around the work device based on the location of the work device of the construction equipment (10). Here, a certain range based on the location of the bucket, arm, and boom, which have been repeatedly mentioned as examples of the work device, may be additionally set as a collision risk range. For example, a critical distance of 10m and a range of 1m to the left and right of the work device may be limited as an additional risk range.

[0141] Meanwhile, the distance range values ​​given here as examples are merely illustrative and, of course, subject to change.

[0142] As described above, according to at least one embodiment of the present invention, sensor data in which a plurality of radars and a plurality of cameras detect an object are fused together, and the measured distance of the radar is reflected in the depth information of the camera and the installation position and angle of each sensor is reflected to align the coordinates, thereby making it possible to accurately detect the position and distance of an object.

[0143] In addition, according to at least one embodiment of the present invention, by setting the area with the highest risk of collision with an object during operation as a region of interest based on the distance and / or coordinates detected by the radar, and excluding sensor data of objects outside the said region of interest from the data fusion target between sensors, it becomes possible for construction equipment to detect hazardous objects quickly and accurately.

[0144] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0145] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

1. A method for improving the detection accuracy of an object by fusing multiple sensor data, performed by a construction equipment safety system, wherein A data collection step for collecting sensor data that detects an object from a sensor unit including a camera and a radar; and A coordinate fusion step for fusing coordinates for sensor data of a camera and a radar included in the sensor unit; The above coordinate fusion step is, Correction is performed on the sensor data collected in the above data collection step to align position coordinate data based on the installation positions of the camera and the radar, method.

2. In Paragraph 1, The above coordinate fusion step is, A radar coordinate generation step for generating three-dimensional radar coordinate data for the position of an object using distance data and azimuth data to the object measured by the radar; A camera coordinate conversion step for converting 2D pixel coordinate data for the position of the object measured by the camera into 3D camera coordinate data; and A coordinate correction step comprising: performing a correction to align the radar coordinate data and the camera coordinate data by reflecting the relative positional difference between the radar and the camera installed on the construction equipment; method.

3. In Paragraph 2, The sensor unit described above includes a plurality of radars installed at different locations, and The above radar coordinate generation step is, A step of generating radar local coordinate data for the position of an object for each radar using distance data and azimuth data to an object measured by each of the plurality of radars; and The method comprises the step of converting each of the above local coordinate data into radar world coordinate data by reflecting the installation position and installation angle of each radar, and determining that objects located at the same coordinate value are duplicated as a single object. method.

4. In Paragraph 2, The sensor unit includes a plurality of cameras installed at different locations, and The above camera coordinate transformation step is, A step of converting the 2D pixel coordinate data of each of the plurality of cameras into camera normalized coordinate data based on the intrinsic parameters of each camera; A step of converting each coordinate value of each of the above-mentioned camera normalized coordinate data into camera local coordinate data by reflecting the distance data measured by the radar; and A step comprising: converting each of the above camera local coordinate data into camera world coordinate data by reflecting the installation position and installation angle of each camera, and determining that objects located at the same coordinate value are duplicated as a single object; method.

5. In Paragraph 2, The above coordinate correction step is, Characterized by performing a transformation on the above-mentioned three-dimensional radar coordinate data that reflects the positional and angular differences in which the camera and the radar are installed on the construction equipment, thereby aligning the coordinates based on the camera. method.

6. In Paragraph 1, The above data collection step Characterized by removing error components of data based on the distance to an object measured by the radar and the intensity of the reflected wave from the object. method.

7. In Paragraph 1, The above coordinate fusion step is, Based on the distance to an object measured by the radar, a Region of Interest (ROI) is set only if the distance is within a preset threshold distance. Characterized by removing radar data outside the aforementioned region of interest from the target of the coordinate fusion. method.

8. In Paragraph 7, The above region of interest is, Characterized by being set only within a certain area around the work device based on the location of the work device of the construction equipment. method.

9. A sensor unit installed on construction equipment and including a camera and radar; and A safety controller that receives sensor data detecting an object from the camera and radar and fuses the coordinates of the sensor data of the camera and radar; is included. The above safety controller is, Characterized by performing a correction to align position coordinate data based on the installation locations of the camera and the radar with respect to the sensor data. Construction Equipment Safety System.

10. In Paragraph 9, The above safety controller is, Based on the distance to an object measured by the radar, a Region of Interest (ROI) is set only if the distance is within a preset threshold distance. Characterized by removing radar data outside the aforementioned region of interest from the target of the coordinate fusion. Construction Equipment Safety System.

11. In Paragraph 10, The above region of interest is, Characterized by being set only within a certain area around the work device based on the location of the work device of the construction equipment. Construction Equipment Safety System.

Images