Calibration system for space recognition device for work machine, and work machine

Abstract

A calibration system SYS comprises: a plurality of space recognition devices (an imaging device S4, distance measurement devices S5L, S5R) that are provided in a crane 100 and are capable of detecting the situation around the crane 100; and a controller 30 that performs calibration among the plurality of space recognition devices. The calibration system SYS also comprises a display device D1 or the like for sending notification to a person present around the crane 100 or inside the crane 100. The controller 30 monitors whether or not calibration data has been collected at a plurality of positions, and sends notification of information indicating that the collection of the calibration data has been completed when the calibration data has been collected at the plurality of positions.

Description

Calibration System for Spatial Recognition Device for Working Machine, and Working Machine

[0009] ,

[0008] , ,

[0001] The present disclosure relates to a calibration system for a spatial recognition device for a working machine and a working machine.

[0002] In recent years, in order to monitor the surroundings of a working machine, there has been a tendency to provide a spatial recognition device (for example, an imaging device, a distance measuring device) on the working machine.

[0003] When a spatial recognition device is provided on a working machine, calibration of the spatial recognition device is required. Patent Document 1 describes a working machine that performs calibration based on the shape of at least a part of the working machine included in the detection range.

[0004] International Publication No. 2023 / 063219 <00000​​​​​​​​​​​​This is a side view showing an example of a crane according to the embodiment. This is a block diagram schematically showing an example of the configuration of a crane according to the embodiment. This is a diagram showing an example of the installation position of the spatial recognition device of the crane and the detection range of the spatial recognition device. This is a block diagram showing an example of the configuration of a calibration system for a spatial recognition device provided on a crane according to the embodiment. This is a first conceptual diagram showing the recognition of a calibration target by the position determination unit. This is a second conceptual diagram showing the recognition of a calibration target by the position determination unit. This is a diagram showing an example in which the calibration range where the detection ranges of the imaging device and the distance measuring device overlap is divided into multiple sections. This is a diagram schematically showing an example of notifying the worker of work completion information. This is a flowchart showing the processing flow of the calibration method according to the first embodiment. This is a diagram showing each section of the calibration range according to the first modified example. This is a diagram showing each section of the calibration range according to the second modified example. This is a diagram showing each section of the calibration range according to the third modified example. This is a diagram showing each section of the calibration range according to the fourth modified example. This is a diagram explaining the operation of the calibration system during calibration according to the second embodiment. This is a flowchart showing the processing flow of the calibration method according to the second embodiment. This is a diagram explaining the operation of the calibration system during calibration according to the third embodiment. This is the first diagram showing the operation of the calibration system during calibration according to the fourth embodiment. This is the second diagram showing the operation of the calibration system during calibration according to the fourth embodiment. This is a diagram showing other target path information that is regenerated in the calibration system. This is a flowchart showing the processing flow of the calibration method of the calibration system according to the fifth embodiment.

[0010] Embodiments of the present invention will be described below with reference to the drawings. Furthermore, the embodiments described below are illustrative and not limiting to the invention, and not all features or combinations thereof described in the embodiments are necessarily essential to the invention. In addition, identical or corresponding components in each drawing are denoted by the same or corresponding reference numerals, and their descriptions may be omitted.

[0011] <Overall Configuration of Crane 100> As an example of a work machine according to the embodiment of this disclosure, a mobile crane 100 shown in Figure 1 will be described. Figure 1 is a side view showing an example of the crane 100 according to the embodiment. In the following, the front-rear direction, left-right direction, and up-down direction of the crane 100 will be described along the front-rear direction, left-right direction, and up-down direction as seen from the perspective of the person riding in the crane 100 (hereinafter also referred to as the operator).

[0012] The crane 100 according to this embodiment is a so-called mobile crawler crane. This crane 100 includes a self-propelled crawler-type lower traveling body 1, an upper rotating body 3 that is rotatably mounted on the lower traveling body 1, and an attachment AT that is pivotably mounted on the front side of the upper rotating body 3.

[0013] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1L and 1R. The lower traveling body 1 moves the crane 100 by hydraulically driving each crawler with a left-side travel hydraulic motor 1ML and a right-side travel hydraulic motor 1MR (see Figure 2).

[0014] The upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by the rotating hydraulic motor 2M (see Figure 2).

[0015] The upper slewing body 3 is equipped with a cabin 4, where the operator sits and controls the crane 100, located adjacent to the right side of the attachment AT. The upper slewing body 3 is also equipped with a counterweight 5 at the rear to balance the weight of the attachment AT and the suspended load.

[0016] The attachment AT suspends and transports loads. The attachment AT is composed of a boom 6 which includes a lower boom 61 that is pivotably connected to the boom mounting section of the upper slewing body 3, an intermediate boom 62 connected to the tip of the lower boom 61, and an upper boom 63 connected to the tip of the intermediate boom 62. The boom 6 has sufficient rigidity because it is formed by assembling multiple frames.

[0017] The length of the attachment AT can be changed by increasing or decreasing the number of interconnectable intermediate booms 62 of the boom 6. The attachment AT is also equipped with a backstop 64 on the rear end of the lower boom 61 to restrict the rearward rotation of the boom 6.

[0018] Furthermore, the crane 100 includes a pendant rope 66, an upper spreader 67, a lower spreader 68, a boom luffing wire rope 69, a gantry 71, and a gantry lifting cylinder 72.

[0019] One end of the pendant rope 66 is connected to the rear end of the upper boom 63. The other end of the pendant rope 66 is connected to the upper spreader 67. The upper spreader 67 connects the pendant rope 66 to the boom luffing wire rope 69. The boom luffing wire rope 69 is wound around the boom luffing winch 31 provided on the upper slewing body 3, and is wound in or unwound based on the drive of the boom luffing winch 31.

[0020] The lower spreader 68 is attached to the tip of the gantry 71, which is provided to be able to raise and lower relative to the upper slewing body 3. The gantry lifting cylinder 72 is provided on the upper slewing body 3 and raises and lowers the gantry 71.

[0021] For example, the crane 100 raises the gantry 71 using the gantry lifting cylinder 72 while winding up the boom luffing wire rope 69 with the boom luffing winch 31. This causes the crane 100 to pull the pendant rope 66 via the upper spreader 67, rotating the boom 6 backward and upward. Conversely, the crane 100 can rotate the boom 6 forward and downward by unwinding the boom luffing wire rope 69 with the boom luffing winch 31.

[0022] The crane 100 is equipped with a boom hook 81, a wire rope 82, and a hook overwinding prevention device 83 for holding the suspended load. The boom hook 81 is suspended from the wire rope 82 via a hook bracket 811. In other words, the boom hook 81 constitutes the lower end of the wire rope 82. The hook bracket 811 has a pulley (not shown) inside through which the wire rope 82 is passed.

[0023] One end of the wire rope 82 is fixed to a fixing part provided at the tip of the boom 6. This wire rope 82 extends downward to the hook bracket 811 of the boom hook 81, and then folds back from the hook bracket 811 and extends upward. Furthermore, the wire rope 82 is stretched over a point sheave 651 provided at the tip of the boom 6 and extends to the rear side of the boom 6, and is wound around the front winch 32 provided on the upper slewing body 3 from the rear side of the tip of the boom 6. In addition, a hook overwinding prevention device 83 is provided on the wire rope 82 and defines the upper limit of the boom hook 81.

[0024] The crane 100 can raise the boom hook 81 and lift a load by winding up the wire rope 82 with the front winch 32. Conversely, the crane 100 can lower the boom hook 81 and lower a load by unwinding the wire rope 82 with the front winch 32.

[0025] Next, the configuration of the drive system and control system of the crane 100 will be described with reference to Figure 2. Figure 2 is a schematic block diagram showing an example of the configuration of the crane 100 according to this embodiment.

[0026] <Hydraulic Drive System> The hydraulic drive system of the crane 100 according to this embodiment includes hydraulic actuators HA that hydraulically drive the lower traveling body 1 (left and right crawlers), the upper slewing body 3, and the attachment AT, respectively. These hydraulic actuators HA include travel hydraulic motors 1ML, 1MR, slewing hydraulic motor 2M, boom luffing hydraulic motor 31M, and front winch hydraulic motor 32M, etc.

[0027] The slewing hydraulic motor 2M is an actuator for slewing the upper slewing body 3 relative to the lower traveling body 1. The boom luffing hydraulic motor 31M is an actuator for operating the boom luffing winch 31. The front winch hydraulic motor 32M is an actuator for operating the front winch 32.

[0028] Furthermore, the hydraulic drive system of the crane 100 includes an engine 11, a main pump 14, a pilot pump 15, a control valve unit 17, and a regulator 18.

[0029] Engine 11 is the prime mover and the main power source in the hydraulic drive system. Engine 11 is, for example, a diesel engine that uses light oil as fuel. Engine 11 is mounted, for example, at the rear of the upper slewing body 3. Under the control of the controller 30, which will be described later, engine 11 rotates at a constant speed at a preset target speed and drives the main pump 14 and the pilot pump 15.

[0030] The main pump 14 supplies hydraulic fluid to the control valve unit 17 through a high-pressure hydraulic line. The main pump 14 is mounted at the rear of the upper slewing body 3, for example, similar to the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and under the control of the controller 30, the piston stroke length is adjusted by adjusting the tilt angle of the swash plate by the regulator 18, thereby controlling the discharge flow rate (discharge pressure) of the hydraulic fluid.

[0031] The control valve unit 17 is a hydraulic control device that controls the hydraulic actuators HA in response to the operator's operation of the operating device 38, the content of remote operation, or operation commands related to the automatic operation function output from the controller 30. The control valve unit 17 is mounted, for example, in the center of the upper rotating body 3. The control valve unit 17 is connected to the main pump 14 via a high-pressure hydraulic line and selectively supplies hydraulic fluid supplied from the main pump 14 to each hydraulic actuator in response to the operator's operation or operation commands output from the controller 30. Specifically, the control valve unit 17 includes a plurality of control valves (e.g., directional control valves) that control the flow rate and direction of the hydraulic fluid supplied from the main pump 14 to each hydraulic actuator HA.

[0032] <Operation System> The operation system of the crane 100 includes a pilot pump 15, a controller 30, a proportional valve 29, an operating device 38, and an operation sensor 39.

[0033] The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed-displacement hydraulic pump. The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic fluid discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic fluid is supplied to the various hydraulic devices as pilot pressure.

[0034] The operating device 38 is located near the cockpit of the cabin 4 and is used by the operator to perform various operations on the crane 100. The operating device 38 includes pedal devices and lever devices for operating each of the hydraulic actuators HA.

[0035] For example, the operating device 38 is electrically operated. The operating sensor 39 detects the direction and amount of operation of the operating device 38 by the operator and outputs an operating signal corresponding to each operated actuator to the controller 30.

[0036] The controller 30 then outputs a control command to the proportional valve 29 that corresponds to the content of the operation signal, that is, a control signal corresponding to the operation of the operating device 38. As a result, the proportional valve 29 inputs a pilot pressure corresponding to the operation of the operating device 38 to the control valve unit 17, and the control valve unit 17 can drive each hydraulic actuator HA according to the operation of the operating device 38. The control valve (directional control valve) that drives each hydraulic actuator built into the control valve unit 17 may be of the electromagnetic solenoid type. In this case, the operation signal output from the operating device 38 may be directly input to the control valve unit 17 (electromagnetic solenoid type control valve).

[0037] A proportional valve 29 is provided for each hydraulic actuator HA that is operated by the operating device 38. The proportional valve 29 is located in the pipeline connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured to change the flow area of ​​the pipeline. The proportional valve 29 operates in accordance with the control command output by the controller 30. Therefore, the controller 30 can supply the hydraulic fluid discharged by the pilot pump 15 to the control valves of each hydraulic actuator HA provided in the control valve unit 17 via the proportional valve 29, independently of the operator's operation of the operating device 38.

[0038] <User Interface System> The user interface system of the crane 100 includes an operating device 38, an operating sensor 39, a display device D1, and an input device D2.

[0039] The display device D1 outputs various information to the operator of the crane 100 inside the cabin 4. The display device D1 is installed in a location easily visible to the operator seated inside the cabin 4 and is a device for outputting various information in a visual manner, such as a liquid crystal display or an organic EL (electroluminescence) display.

[0040] The input device D2 is located within close proximity to the operator seated in the cabin 4 and receives various inputs from the operator. The input signals received by the input device D2 are taken up by the controller 30. For example, the input device D2 may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operating device 38 (lever device), etc. Alternatively, the input device D2 may be a voice input device that receives voice input from the operator. A voice input device may include, for example, a microphone. Or, the input device D2 may be a gesture input device that receives gesture input from the operator. A gesture input device may include, for example, an imaging device (indoor camera) installed in the cabin 4.

[0041] <Communication System> The communication system of the crane 100 includes a communication device T1 capable of communicating with external devices. The communication device T1 is connected to a communication line and communicates with a device provided separately from the crane 100. An example of a device provided separately from the crane 100 is a portable communication terminal 800 carried by a worker at the work site. The communication device T1 may also include a mobile communication module compliant with standards such as 4G (4th Generation) or 5G (5th Generation). For example, the communication device T1 may also include a satellite communication module, a Wi-Fi® communication module, or a Bluetooth® communication module.

[0042] <Control System> The control system of the crane 100 includes, for example, a slewing sensor S1, a boom luffing sensor S2, a length sensor S3, an upper slewing body positioning device PM, a storage device ST, and a controller 30. Furthermore, the crane 100 according to this embodiment includes an imaging device S4 and distance measuring devices S5L and S5R as a spatial recognition device capable of acquiring external surrounding conditions. The control system of the crane 100 also includes a speaker SP (sound output device) that outputs sound information (voice, buzzer sound, etc.) to the outside, and a light source L installed outside the crane 100.

[0043] The slewing sensor S1 outputs information regarding the slewing of the upper slewing body 3. The slewing sensor S1 detects, for example, the slewing angular velocity of the upper slewing body 3 with respect to the lower traveling body 1. Further, the slewing sensor S1 detects the slewing angle. The slewing sensor S1 may be, for example, a gyro sensor, a resolver, a rotary encoder, or an IMU (Inertial Measurement Unit). A detection signal corresponding to the slewing angle or slewing angular velocity of the upper slewing body 3 by the slewing sensor S1 is taken into the controller 30.

[0044] The boom hoisting sensor S2 outputs information regarding the hoisting of the boom 6. The boom hoisting sensor S2 detects, for example, the hoisting angle (tilt angle) of the boom 6. The boom hoisting sensor S2 may be, for example, a gyro sensor or an IMU. A detection signal corresponding to the hoisting angle of the boom 6 by the boom hoisting sensor S2 is taken into the controller 30.

[0045] The length sensor S3 outputs information regarding the length of the wire rope 82 for suspending the suspended load with the boom hook 81. The length sensor S3 is, for example, an encoder provided in the front winch 32, and detects the length of the wire rope 82 unwound from the front winch 32.

[0046] The upper slewing body positioning device PM measures the position of the upper slewing body 3. The upper slewing body positioning device PM is, for example, a GNSS (Global Navigation Satellite System) positioning device, and detects the position and orientation of the upper slewing body 3. A detection signal corresponding to the position and orientation of the upper slewing body 3 is taken into the controller 30. The function of detecting the orientation of the upper slewing body 3 may be realized by an azimuth sensor attached to the upper slewing body 3. The upper slewing body positioning device PM measures the current position of the crane 100 in a set reference coordinate system.

[0047] The reference coordinate system is, for example, a World Geodetic System capable of specifying positions on the earth. The World Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the origin at the center of gravity of the earth, the X-axis in the direction of the intersection of the Greenwich meridian and the equator, the Y-axis in the direction of 90 degrees east longitude, and the Z-axis in the direction of the North Pole.

[0048] The peripheral recognition device ES acquires information on the surroundings outside the crane 100. This peripheral recognition device ES is composed of, for example, one or a combination of an imaging device such as a camera, a LiDAR (Light Detection and Ranging), an ultrasonic sensor, or other optical sensors. The information acquired by the peripheral recognition device ES is converted into information recognizable by the controller 30 through well-known image processing or the like, and is taken into the controller 30. The peripheral recognition device ES may apply the same device as the space recognition device described later.

[0049] The storage device ST is, for example, a readable and writable non-volatile storage medium. For this storage device ST, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive) can be applied.

[0050] The controller 30 controls the operations of the respective drive units provided in the crane 100. The functions of the controller 30 may be realized by any hardware, or any combination of hardware and software, etc. For example, the controller 30 is centered around a computer including a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory), a non-volatile auxiliary storage device such as a ROM (Read Only Memory), and various input / output interface devices. The controller 30 realizes various functions by loading a program installed in the auxiliary storage device into the memory device and executing it on the CPU.

[0051] For example, the controller 30 controls the operation of the hydraulic actuator HA of the crane 100 based on the operation of the operating device 38, with the proportional valve 29 as the control target. The controller 30 may also provide operation support to assist in transporting the suspended load to the target position on the ground surface of the crane 100 (for example, the plane of a two-dimensional coordinate system of XY coordinates). Operation support for the crane 100 may include fully automatic operation that controls the operation of the entire crane 100, semi-automatic operation that controls some of the operations of the crane 100, and a guidance function that displays the operation of the crane 100 on the display device D1 (or outputs it from the speaker).

[0052] As an example, in the crane 100, while the operator controls the hoisting and lowering of the suspended load, the controller 30 automatically controls the slewing and luffing of the crane 100 in a semi-automatic operation. In other words, the controller 30 controls only the slewing of the upper slewing body 3 and the luffing of the boom 6 to move the boom hook 81 to the target position in the two-dimensional coordinate system. In this semi-automatic operation, the controller 30 controls the slewing speed of the upper slewing body 3 and the luffing speed of the boom 6 to suppress the swaying of the suspended load. For example, the controller 30 outputs current to the proportional valve 29 to apply an appropriate pilot pressure to the control valve unit 17. As a result, the crane 100 can automatically control the slewing hydraulic motor 2M and the boom luffing hydraulic motor 31M.

[0053] Figure 3 shows an example of the installation position of the spatial recognition device for the crane 100 and the detection range of the spatial recognition device. One of the spatial recognition devices, the imaging device S4, is attached, for example, to the upper rear end of the counterweight 5 and images the area behind the crane 100. In this embodiment, the crane 100 is configured to have one imaging device S4, but multiple imaging devices S4 may be provided. The imaging device S4 may also image the area in front of or to the side of the crane 100. Furthermore, the imaging device S4 is not limited to being installed on the upper rear end of the counterweight 5, but may be installed in a position that can monitor the area around the crane 100. The installation position of the imaging device S4 may be, for example, the upper left side, upper right side, or upper front part of the counterweight 5.

[0054] The detection range 1001 of the imaging device S4 is, for example, the entire rear of the crane 100, including a part of the left side and a part of the right side. The detection range 1001 of the imaging device S4 is set to a range of approximately 120° to 180° with the imaging device S4 as the base point. For example, the imaging device S4 is a monocular wide-angle camera with a very wide field of view. Alternatively, the imaging device S4 may be a stereo camera or a depth image camera, etc. The imaging information detected by the imaging device S4 is taken up by the controller 30.

[0055] The distance measuring devices S5L and S5R are installed at the lower rear end of the counterweight 5 and detect the distance to objects present around the crane 100. Distance measuring device S5L is mounted on the lower rear end of the counterweight 5, closer to the left side. The detection range 1002 of distance measuring device S5L is the range from the rear to the left of the crane 100. Distance measuring device S5R is mounted on the lower rear end of the counterweight 5, closer to the right side. The detection range 1003 of distance measuring device S5R is the range from the rear to the right of the crane 100. The detection ranges 1002 and 1003 of distance measuring devices S5L and S5R are, for example, in the range of approximately 120° to 270°. In this embodiment, two distance measuring devices S5L and S5R are provided, but the system is not limited to this configuration; it may also have one or three or more devices. Furthermore, the distance measuring device may be provided on the upper surface of the counterweight 5 or on the front side of the upper rotating body 3 of the crane 100 in order to detect objects located ahead.

[0056] The distance measuring devices S5L and S5R are, for example, scanning LiDARs, and a three-dimensional laser scanner capable of scanning the direction of infrared laser irradiation in the vertical and horizontal directions can be applied. In this case, the distance measuring devices S5L and S5R irradiate infrared light in a certain direction and receive reflected light from an object in that direction, thereby acquiring information about objects around the crane 100 as received light information. Alternatively, the distance measuring devices S5L and S5R may be so-called flash-type LiDARs that irradiate infrared light over a wide three-dimensional area from a light-emitting module and capture the reflected light (infrared light) with a three-dimensional distance image element.

[0057] The imaging device S4, and the distance measuring devices S5L and S5R are each preferably mounted on the counterweight 5 or the upper part of the upper rotating body 3 such that their optical axes are pointed diagonally downward. This allows the imaging device S4, and the distance measuring devices S5L and S5R to detect objects in vertical detection ranges 1001, 1002, and 1003, which include the area from the ground near the crane 100 to the area far from the crane 100.

[0058] An overlapping region OL is set in the detection ranges 1001, 1002, and 1003 of the imaging device S4 and the distance measuring devices S5L and S5R, respectively. If an object is present in the overlapping region OL, multiple devices will detect the same object. Therefore, calibration is necessary between the imaging device S4 and the distance measuring devices S5L and S5R to prevent false detection of objects in the overlapping detection ranges (for example, to prevent the same object from being detected as different objects by each device). In other words, calibration according to this embodiment refers to the process of defining the positional relationship (coordinate system) of the detection ranges between the multiple spatial recognition devices. By calibrating the multiple spatial recognition devices, the controller 30 can suppress false recognition, such as double recognition of an object, when the same object is detected by each of the multiple spatial recognition devices.

[0059] In particular, the crane 100 is assembled according to the site. Also, the number of weights stacked on the counterweight 5 may differ from site to site. Therefore, the relationship between the installation positions of the multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) installed on the crane 100 will also change each time it is assembled on site or for each type of work. For this reason, the multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) of the crane 100 need to be calibrated each time it is assembled on site, and a method is needed that allows the worker W to easily perform the calibration on site.

[0060] <Calibration of the Spatial Recognition Device According to the First Embodiment> The following describes a method for calibrating the spatial recognition device (imaging device S4, distance measuring devices S5L, S5R) according to the first embodiment. In the following, the calibration between the imaging device S4, distance measuring devices S5L, and S5R described above as multiple spatial recognition devices will be described. However, the calibration of the spatial recognition device is not limited to this, and may also be performed when multiple imaging devices S4 are combined, or when multiple distance measuring devices S5L and S5R are combined, etc.

[0061] Alternatively, the spatial recognition device may be any device capable of acquiring information about the surrounding conditions of the crane 100, and may be a device other than the imaging device S4 and the distance measuring devices S5L and S5R. Other examples of spatial recognition devices include other object detection devices such as ultrasonic sensors, millimeter-wave radar, LiDAR, and infrared sensors. When using millimeter-wave radar, ultrasonic sensors, or laser radar as the spatial recognition device, multiple signals (such as laser light) may be transmitted to an object, and the distance and direction of the object may be detected from the reflected signals by receiving the reflected signals.

[0062] Furthermore, the embodiment describes an example in which a person is used as a calibration target. Typically, a marker with "X" or the like is often used as a calibration target. However, when the crane 100 is assembled at the work site, it becomes necessary to bring the calibration target to the work site. If the calibration target has to be brought in every time work is performed at the work site, the burden on the worker W becomes significant. Therefore, by using a person present at the work site as a calibration target, the burden on the worker W and the time loss associated with preparing the calibration target can be reduced. However, conventional calibration targets such as markers may also be used for calibration.

[0063] During the calibration of the imaging device S4 and the distance measuring devices S5L and S5R, the operator W, who is the calibration target, is moved within the overlapping region OL where the detection ranges 1001, 1002, and 1003 overlap. When the imaging device S4, distance measuring devices S5L, and S5R detect this operator W and transmit the information to the controller 30, the controller 30 processes the detection information from the imaging device S4 and the detection information from the distance measuring devices S5L and S5R as calibration data. In other words, calibration data is information acquired by the spatial recognition device, and it is data from which the position coordinates of the object (calibration target) used for calibration can be extracted from the detection information. The controller 30 performs calibration processing to align the coordinate positions of the different devices using the position coordinates of the same operator W (calibration target) extracted from the calibration data of the imaging device S4, distance measuring devices S5L, and S5R. As a result, the controller 30 can calibrate so that the detection information from the imaging device S4 and the detection information from the distance measuring devices S5L and S5R recognize the operator W as the same entity.

[0064] However, simply integrating detection information based on an operator W at only one location within the overlapping detection ranges 1001, 1002, and 1003 is insufficient as calibration data, as it fails to adjust the image of the entire area where the detection ranges 1001, 1002, and 1003 overlap. In other words, accurate calibration can be achieved by having the operator W move to many parts of the overlapping area OL of the detection ranges 1001, 1002, and 1003 and detecting and processing the information at those locations. Thus, when operator W moves to many parts of the overlapping area OL to acquire calibration data during the calibration of multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R), the time required for calibration work increases. If the operator finishes the calibration work with insufficient calibration data, the accuracy of the calibration will decrease, and in some cases the calibration work may have to be redone, resulting in poor work efficiency. Therefore, the work machine (crane 100) according to this disclosure aims to improve work efficiency by enabling notification of the status of the calibration work.

[0065] <Block Configuration of Each Device for Calibration> Figure 4 is a block diagram showing an example of the configuration of a spatial recognition device calibration system SYS installed on a crane 100 according to the first embodiment. The spatial recognition device calibration system SYS includes the crane 100 and a portable communication terminal 800 connected via a communication line NW. However, the calibration system SYS may be implemented using only the configuration of the crane 100 without including the portable communication terminal 800.

[0066] The portable communication terminal 800 is a communication terminal carried by the worker W who is the calibration target. The controller 30 of the crane 100 can notify information during calibration via the portable communication terminal 800. In this embodiment, the portable communication terminal 800 is not limited to the worker W who is the calibration target; another person may hold the portable communication terminal 800 and give instructions for the work.

[0067] The mobile communication terminal 800 comprises a display device 801, an input device 802, a communication device T2, and a controller 830. The display device 801 is, for example, a liquid crystal panel provided on the mobile communication terminal 800, which displays various information images under the control of the controller 830. The input device 802 is an interface for the operator W to input information, and for example, a touch panel provided on the display device 801 can be applied.

[0068] The communication device T2 connects to the communication line NW and communicates with the crane 100, etc. The communication device T2 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) or 5G (5th Generation). The communication device T2 may also include, for example, a satellite communication module. Furthermore, the communication device T2 may include, for example, a Wi-Fi® communication module or a Bluetooth® communication module.

[0069] The controller 830 includes, for example, a CPU (or GPU), memory, and an input / output interface, and performs various controls by executing a program stored in memory via the CPU. In the calibration of the spatial recognition device, a display control unit 831 and a selection unit 832 are constructed inside the controller 830.

[0070] The display control unit 831 controls the display device 801 to display various information. For example, the display control unit 831 displays a list of multiple spatial recognition devices (e.g., imaging device S4, distance measuring devices S5L, S5R) installed on the crane 100 in a selectable format.

[0071] The selection unit 832 selects a plurality of spatial recognition devices to be calibrated. For example, the selection unit 832 receives a spatial recognition device selected by the operator W via the input device 802 from a list of a plurality of spatial recognition devices (e.g., imaging device S4, distance measuring devices S5L, S5R) displayed on the display device 801. Alternatively, the selection unit 832 may automatically select a plurality of spatial recognition devices. The selection unit 832 then transmits information indicating the selected plurality of spatial recognition devices to the crane 100 via the communication device T2. The selection unit 832 may be located within the controller 30 of the crane 100.

[0072] Meanwhile, the controller 30 of the crane 100 internally constructs a first object detection unit 301, a second object detection unit 302, a target extraction unit 303, a calibration status recognition unit 304, an output control unit 305, a feature extraction unit 306, and a calibration unit 307 for the calibration of the spatial recognition device.

[0073] The first object detection unit 301 extracts objects located within the detection ranges 1002 and 1003 of the distance measuring devices S5L and S5R based on the detection information from the distance measuring devices S5L and S5R. Any known method may be used to extract objects from the detection information of the distance measuring devices S5L and S5R. For example, the first object detection unit 301 may generate a distance image (with different colors depending on the distance) based on the distance information shown in the point cloud data of the detection information, and extract objects from the visualized distance image.

[0074] The second object detection unit 302 extracts objects present in the detection range 1001 of the imaging device S4 based on the detection information from the imaging device S4. Any known method may be arbitrarily employed for extracting objects from the detection information of the imaging device S4. For example, the second object detection unit 302 may identify the shape of an object by contour extraction from the detection information and then detect the object based on the identified shape.

[0075] The target extraction unit 303 extracts the region where the operator W, which will be the calibration target, is located from the objects detected by the first object detection unit 301. The target extraction unit 303 also extracts the region where the operator W, which will be the calibration target, is located from the objects detected by the second object detection unit 302.

[0076] The calibration status recognition unit 304 recognizes the position (coordinates) of the calibration target for each of the multiple spatial recognition devices performing calibration, based on the region extracted by the target extraction unit 303. The position of the calibration target can be recognized in a unified coordinate system, for example, by setting a reference spatial recognition device and aligning the calibration targets of the other spatial recognition devices with the calibration target of the coordinate system of the reference spatial recognition device. An example of a reference spatial recognition device is the imaging device S4 located midway in the width direction between the distance measuring devices S5L and S5R.

[0077] Figure 5A is a first conceptual diagram showing the recognition of the calibration target by the calibration status recognition unit 304. Figure 5B is a second conceptual diagram showing the recognition of the calibration target by the calibration status recognition unit 304. In Figures 5A and 5B, image 1101 is an image based on detection information from the imaging device S4, image 1102 is an image based on detection information from the distance measuring device S5L, and image 1103 is an image based on detection information from the distance measuring device S5R.

[0078] In calibration targeting worker W, the entire worker W is detected and used as calibration data. However, calibration may be performed using only a portion of worker W.

[0079] As shown in Figure 5A, if an extracted image WC of the entire worker W exists in all three images 1101, 1102, and 1103, then the worker W is located in the overlapping region OL described above. Therefore, the calibration status recognition unit 304 can integrate the extracted images WC of the worker W in the three images 1101, 1102, and 1103 and place each extracted image WC at a single reference coordinate (in this embodiment, the imaging device S4).

[0080] On the other hand, as shown in Figure 5B, in some of the three images 1101, 1102, and 1103 (for example, images 1101 and 1102), the entire worker W may not be included. In this case, the worker W is located in a position shifted from the overlapping area OL, and the calibration status recognition unit 304 determines that it should not be used as calibration data for the spatial recognition device.

[0081] Thus, in the calibration of the spatial recognition device, it is important that an operator is present in the overlapping region OL of the detection ranges 1001, 1002, and 1003 of the imaging device S4 and the distance measuring devices S5L and S5R. However, calibration data with only the operator present at a single point (for example, the center) in the overlapping region OL is insufficient for calibration, as it does not reveal the overlapping region OL of the detection ranges 1001, 1002, and 1003 of multiple spatial recognition devices. Therefore, in the calibration of the spatial recognition device according to the first embodiment, as shown in Figure 6, the calibration range 2000 (overlapping region OL) where the detection ranges 1001, 1002, and 1003 of the imaging device S4 and the distance measuring devices S5L and S5R overlap is divided into multiple sections 2001, and calibration data is acquired for each section 2001. Figure 6 shows an example in which the calibration range 2000, where the detection ranges 1001, 1002, and 1003 of the imaging device S4, distance measuring devices S5L, and S5R overlap, is divided into multiple sections 2001.

[0082] The calibration range 2000 has multiple rectangular sections 2001, for example, by being divided in a matrix. Figure 6 shows an example in which the multiple sections 2001 are arranged in four vertical columns and four horizontal rows, resulting in a total of 16 sections 2001. Of course, the number of divisions of the calibration range 2000 is not limited to this and can be set arbitrarily.

[0083] In calibrating a spatial recognition device, it is preferable to obtain calibration data evenly across multiple sections 2001. This is because the accuracy of the calibration can be improved as the amount of calibration data in the calibration range 2000 increases. However, in calibrating a spatial recognition device, it is not necessary to acquire calibration data for all sections 2001; calibration can be completed without acquiring calibration data for some sections 2001. This is because setting the device to acquire calibration data for all sections 2001 would lengthen the calibration work period.

[0084] Therefore, Figure 6 shows an example of performing calibration of the spatial recognition device by acquiring calibration data for 14 of the 16 sections 2001. In the calibration of the spatial recognition device, the calibration system SYS detects the worker W moving within the calibration range 2000 (overlapping area OL) using the imaging device S4, distance measuring devices S5L and S5R, and acquires calibration data for each of the multiple sections 2001.

[0085] In this case, worker W may move arbitrarily within the area assumed to be the overlapping region OL. After calibration begins, the imaging device S4, distance measuring devices S5L, and S5R automatically repeat detection of the moving worker W and transmit the detection information to the controller 30 as it occurs, thereby accumulating calibration data. The accumulated detection information includes worker W's information, device identification information, detection time, etc. Figure 6 shows an example where worker W randomly walks through the area assumed to be the overlapping region OL. It can be seen that the trajectory WL of worker W, walked randomly, passes through multiple designated sections 2001. In this way, the controller 30 can use the detection information of worker W detected by the imaging device S4, distance measuring devices S5L, and S5R as worker W passes through each section 2001 as calibration data to calibrate the spatial recognition device.

[0086] For example, when the controller 30 acquires calibration data when a worker passes through a designated section 2001, it recognizes that section 2001 as a section 3000 for which calibration data has been acquired. In other words, a section 3000 for which calibration data has been acquired is information about section 2001 that indicates calibration data can be obtained from the detection information of the imaging device S4, the distance measuring devices S5L and S5R (in other words, information related to the coordinates from which calibration data was obtained). By accumulating sections 3000 for which calibration data has been acquired as the worker W moves, it becomes possible to identify which of the divided sections 2001 has had calibration data acquired. Figure 6 shows an example in which 14 out of 16 sections 2001 have become sections 3000 for which calibration data has been acquired.

[0087] However, in the calibration of the spatial recognition device, the setting of the calibration range 2000 and the divided sections 2001, and the recognition of sections 3000 for which calibration data has been acquired, are based on the processing of the controller 30. If the operator W performing the calibration work does not receive notification from the crane 100 or the mobile communication terminal 800, there is no way for the operator W to determine whether the acquisition of calibration data has been completed. In this case, the operator W may complete the calibration work even though the calibration data is insufficient. If the calibration data for the spatial recognition device is insufficient, the accuracy of the calibration will decrease, and in some cases, the calibration may have to be repeated.

[0088] Therefore, in the calibration of the spatial recognition device according to the first embodiment, the operator W is notified that sufficient calibration data has been collected for each section 2001 of the calibration range 2000 and that the acquisition of calibration data has been completed. The information that the acquisition of calibration data has been completed means that the operator W can finish the calibration work of walking around while being aware of the overlapping area OL (therefore, this will also be referred to as work completion information below). By receiving the notification provided by the calibration system SYS, the operator W can clearly recognize that their calibration work has been completed.

[0089] As shown in Figure 4, when the output control unit 305 receives information from the calibration status recognition unit 304 that sufficient calibration data has been collected for each section 2001 of the calibration range 2000, it notifies the worker W of the completion of work through the various configurations of the crane 100. Note that the worker to whom the completion of work information is notified is not limited to the worker W who is actually moving through the overlapping area OL. This worker may be the operator of the crane 100 who is in the cabin 4, or another person who is in the vicinity of the crane 100 (for example, a site supervisor). This is because the operator or another person can recognize that calibration is possible and then communicate this to the worker W who is actually moving.

[0090] Therefore, examples of notification units for the crane 100 that notify of work completion information include a display device D1, a speaker SP, a light source L, etc. The operator of the crane 100 can recognize the work completion information via the display device D1. The crane 100 can also directly notify the worker W of the work completion information via the speaker SP or the light source L. Furthermore, the output control unit 305 may transmit the work completion information to the portable communication terminal 800, which is the notification device, via the communication device T1. The portable communication terminal 800 can notify the worker W of the work completion information by displaying it on the display device 801 (or outputting it from the sound output device, or causing the portable communication terminal 800 to vibrate).

[0091] Figure 7 schematically shows an example of notifying worker W of work completion information. For example, the crane 100 can output work completion information to worker W as audio information via speaker SP. An example of audio information is, "Calibration data acquisition is complete." Alternatively, the crane 100 may notify worker W of work completion information by lighting (or flashing) a light source L in a predetermined color.

[0092] Alternatively, as described above, if the crane 100 transmits work completion information to the mobile communication terminal 800, the mobile communication terminal 800 can output voice information such as "Calibration data acquisition complete." Alternatively, the mobile communication terminal 800 may display image information such as "Calibration data acquisition complete" on the display device 801. Furthermore, the mobile communication terminal 800 may notify the worker W of the work completion information by outputting a predetermined buzzer sound or vibrating.

[0093] Returning to Figure 4, the feature extraction unit 306 of the controller 30 uses the accumulated calibration data to extract characteristic parts and feature quantities of the calibration targets in each of the multiple spatial recognition devices. The calibration targets can be identified from the region where the calibration targets extracted by the target extraction unit 303 exist.

[0094] The calibration unit 307 performs calibration between multiple spatial recognition devices based on the characteristic parts and feature quantities of the calibration target extracted by the feature extraction unit 306. Since a well-known method can be used for this calibration, a detailed explanation will be omitted.

[0095] The crane 100 according to the first embodiment is basically configured as described above, and its operation (calibration method) will be explained below with reference to Figure 8. Figure 8 is a flowchart showing the processing flow of the calibration method according to the first embodiment.

[0096] In the calibration method, the controller 30 of the crane 100 controls various devices to execute steps S101 to S105 shown in Figure 8.

[0097] When the calibration method is started, the imaging device S4 and the distance measuring devices S5L and S5R first image the worker W in their respective detection ranges 1001, 1002, and 1003 (step S101). At this time, the worker W may freely walk around in the area estimated to be the calibration range 2000 (overlapping area OL). If the worker W is in the calibration range 2000, the imaging device S4 and the distance measuring devices S5L and S5R can each acquire detection information including the entire worker W. When acquiring each piece of detection information, the first object detection unit 301 and the second object detection unit 302 of the controller 30 extract the object in their respective detection information.

[0098] Then, the target extraction unit 303 of the controller 30 extracts the operator W, which is the calibration target, from the detection information of the imaging device S4 at regular intervals and stores its coordinates, and also extracts the operator W from the detection information of the distance measuring devices S5L and S5R at regular intervals and stores its coordinates (step S102).

[0099] When the coordinates of the operator W of each spatial recognition device are stored, the calibration status recognition unit 304 of the controller 30 sets section 2001 of the calibration range 2000 as section 3000 for which calibration data has been acquired, in the coordinates of the imaging device S4, which is the reference device (step S103).

[0100] The calibration status recognition unit 304 then determines whether the number of sections 3000 for which calibration data has been acquired has reached a threshold (step S104). In this case, the threshold may be automatically set internally based on the calibration accuracy and the number of sections 2001 in the calibration range 2000. In the example in Figure 6, based on the existence of 16 sections 2001, 14 sections 2001 are set as the threshold. If the number of sections 3000 for which calibration data has been acquired is below the threshold (step S104: NO), the process returns to step S101 and the same process is repeated thereafter. On the other hand, if the number of sections 3000 for which calibration data has been acquired is above the threshold (step S104: YES), the process proceeds to step S105.

[0101] In step S105, the output control unit 305 of the controller 30 notifies the worker W of the completion of work. As described above, the completion of work information is notified via a notification unit (at least one of the display device D1, speaker SP, light source L, or portable communication terminal 800). By recognizing this completion of work information through the notification unit, the worker W can determine that their calibration work is complete.

[0102] Furthermore, the feature extraction unit 306 and calibration unit 307 of the controller 30 automatically start a calibration process to align the position of the calibration target (operator W) in the imaging device S4, distance measuring devices S5L, and S5R using the calibration data of each section 3000 for which calibration data has been acquired. As a result, the calibration system SYS can perform calibration of multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, and S5R) with high accuracy.

[0103] As described above, the calibration system SYS according to the first embodiment notifies the operator W of the completion of work information when the acquisition of calibration data is complete. This allows the operator W to know reliably and promptly that their calibration work has finished and to stop their calibration work. Conversely, if there is no notification of completion of work, the operator will continue the walking motion of the calibration work. Therefore, the calibration system SYS can avoid inconveniences such as insufficient calibration data and having to redo the calibration work in the calibration of the spatial recognition device.

[0104] Furthermore, the calibration system SYS and the work machine (crane 100) of this disclosure are not limited to the embodiments described above and can be modified in various ways. For example, the work machine is not limited to a crane applied to a construction work site, but may also be a crane (low-floor type, gantry type, tower type) installed in a port or the like. Also, the work machine is not limited to a mobile crane, but may also be a fixed crane. Alternatively, the work machine may be an overhead crane or bridge crane with a movable part suspended above a runway. Furthermore, the work machine may be a truck crane equipped with a crane, or an excavator with a crane function that applies a wire rope to the excavator attachment. In addition, the work machine is not limited to a crane, but may also be road machinery including a forklift or asphalt finisher, or construction machinery including an excavator.

[0105] Furthermore, notification of the completion of calibration data acquisition is not limited to the above-mentioned notification. For example, the operator or worker W can be made aware of the completion of calibration data acquisition by displaying the calibration data acquisition log information on the display device D1 of the crane 100 (or the display device 801 of the mobile communication terminal 800). The calibration data acquisition log information may also be image information displaying each section 2001 of the calibration range 2000 and the section 3000 for which calibration data has been acquired.

[0106] Alternatively, for example, the calibration system SYS may recognize the shape (area, etc.) of the area where calibration data is insufficient based on the trajectory WL of the operator W, without dividing the calibration range 2000 into multiple sections 2001, and determine when to complete the acquisition of calibration data. In this case, a notification may be sent to prompt the operator W to move to the area where calibration data is insufficient (see also the fourth embodiment).

[0107] Furthermore, the sections 2002 that divide the calibration range 2000 are not limited to the rectangular frame described above, but may take various forms. For example, other section shapes include those shown in Figures 9A to 9D. Figure 9A shows each section 2002 of the calibration range 2000 according to the first modification. Figure 9B shows each section 2003 of the calibration range 2000 according to the second modification. Figure 9C shows each section 2004 of the calibration range 2000 according to the third modification. Figure 9D shows each section 2005 of the calibration range 2000 according to the fourth modification.

[0108] The multiple sections 2002 in the first modified example shown in Figure 9A are formed in a circular (elliptical) shape and arranged in a matrix. In this way, even when applying the circular sections 2002 to the calibration range 2000, the spatial recognition device can be calibrated by acquiring multiple calibration data for each section 2002 as the operator W moves, similar to the above.

[0109] Furthermore, the multiple compartments 2003 in the second modified example shown in Figure 9B are formed in an elliptical shape with a major axis in the horizontal direction and are arranged in a vertical direction (up and down). Depending on the arrangement of the spatial recognition device, calibration data may be acquired by arranging each compartment 2003 vertically in this manner. For example, if the imaging device S4 and the distance measuring device are installed side by side in the vertical direction, sufficient calibration data can be obtained even if each compartment 2003 is set to be arranged vertically.

[0110] Furthermore, the multiple compartments 2004 in the third modified example shown in Figure 9C are formed in an elliptical shape with a major axis in the vertical direction and are arranged in a horizontal direction (left to right). Depending on the arrangement of the spatial recognition device, calibration data may be acquired by arranging each compartment 2004 side by side in this manner. For example, when multiple distance measuring devices S5R and S5L are installed side by side, sufficient calibration data can be obtained even if each compartment 2003 is set to be arranged side by side.

[0111] Furthermore, the multiple compartments 2005 in the fourth modified example shown in Figure 9D are formed in a honeycomb structure with hexagonal frames arranged without gaps. Even in this case, the controller 30 can acquire multiple calibration data for each compartment 2005 and perform calibration of the spatial recognition device.

[0112] <Calibration of the Spatial Recognition Device According to the Second Embodiment> Figure 10 is a diagram illustrating the operation of the calibration system SYSα according to the second embodiment during calibration. The calibration system SYSα according to the second embodiment differs from the calibration system SYS described above in that, in the calibration of the spatial recognition device, it notifies the operator W of the progress when acquiring calibration data for each section 2001 of the calibration range 2000. Note that other parts of the calibration system SYSα are the same as those of the first embodiment, and a detailed explanation thereof is omitted. The same applies to the third to fifth embodiments thereafter.

[0113] For example, the calibration status recognition unit 304 (Figure 4) of the controller 30 of the crane 100 calculates the ratio of the number of sections 3000 for which calibration data has been acquired to the number of sections (threshold) for which calibration has been completed. In Figure 10, a state is shown where 13 sections 3000 have acquired calibration data for a section 2001 with a threshold of 14, and in this case, 93% is notified as progress information. In other words, the progress information is information that indicates how much of the multiple sections 2001 (locations) have acquired calibration data (data that can detect the position of a predetermined object). The progress information may also notify the operator W of the number of sections that have been completed (14 in Figure 10) and the number of sections for which calibration data has been acquired (13 in Figure 10).

[0114] When the output control unit 305 (Figure 4) acquires progress information from the calibration status recognition unit 304, it transmits the progress information to the mobile communication terminal 800. As a result, the mobile communication terminal 800 displays a progress information image 840 based on the progress information on the display device 801, allowing the worker W to recognize the current status. When notifying the worker W of the progress information, various configurations are available, not limited to displaying it on the display device 801 of the mobile communication terminal 800. For example, as shown in the dashed-dotted callout in Figure 7, voice information ("80% of the data has been acquired," etc.) may be output as progress information from the speaker SP of the crane 100. Alternatively, the crane 100 may notify the worker W of the progress information by flashing the light source L. Or, the mobile communication terminal 800 may notify the worker W of the progress information not limited to displaying it on the display device 801, but also by voice information, a buzzer, vibration, etc.

[0115] Figure 11 is a flowchart showing the processing flow of the calibration method according to the second embodiment. In the calibration method of the calibration system SYSα according to the second embodiment, each step S111 to S113 is basically the same as steps S101 to S103 of the calibration method according to the first embodiment, and a detailed explanation thereof will be omitted.

[0116] In step S114, the controller 30 calculates progress information based on the number of sections 3000 for which calibration data has been acquired and the number of sections for which calibration has been completed at the threshold, and updates the progress information image 840 on the mobile communication terminal 800 by transmitting this progress information. This allows the mobile communication terminal 800 to recognize the progress of the operator W's own calibration work.

[0117] Furthermore, steps S115 to S116 are the same as steps S104 to S105 of the calibration method according to the first embodiment. Therefore, a detailed explanation thereof will be omitted.

[0118] As described above, the calibration system SYSα and calibration method according to the second embodiment can notify the operator W of the progress of the calibration work during the work. This allows the operator to perform the calibration work while being aware of how much calibration data has been collected, thereby improving work efficiency.

[0119] <Calibration of the Spatial Recognition Device According to the Third Embodiment> Figure 12 is a diagram illustrating the operation of the calibration system SYSβ according to the third embodiment during calibration. The calibration system SYSβ according to the third embodiment differs from the above-mentioned calibration systems SYS and SYSα in that, in the calibration of the spatial recognition device, it generates target path information 850 for passing through each section 2001 for the operator W and notifies the operator W.

[0120] For example, before the calibration of the spatial recognition device begins, the calibration status recognition unit 304 of the controller 30 forms the calibration range 2000 and each section 2001 based on the detection information of the reference device, the imaging device S4, and generates target route information 850 based on each section 2001. The output control unit 305 transmits this target route information 850 to the mobile communication terminal 800. This allows the mobile communication terminal 800 to provide the target route information 850 to the worker W.

[0121] For example, the target route information 850 is displayed on the display device 801 along with the calibration range 2000 and each section 2001. In this case, the image captured by the imaging device S4 may be displayed as the background. This allows the worker W to perform the calibration work by roughly recognizing the target route information 850 and the places to walk.

[0122] However, the target route information 850 is reference information, and the worker W does not have to move along the target route information 850. Figure 12 shows the trajectory WL of worker W moving freely without relying on the target route information 850. The calibration system SYSβ can obtain calibration data according to this trajectory WL of worker W.

[0123] <Calibration of the Spatial Recognition Device According to the Fourth Embodiment> Figures 13A and 13B show the operation of the calibration system SYSγ according to the fourth embodiment during calibration. The calibration system SYSγ according to the fourth embodiment differs from the above-mentioned calibration systems SYS, SYSα, and SYSβ in that, in the calibration of the spatial recognition device, it regenerates (updates) target route information 851 according to the section 3000 for which calibration data has been acquired when the operator W moves, and notifies the operator W.

[0124] In other words, as shown in Figure 13A, the calibration system SYSγ provides target route information 850 generated before the start of calibration of the spatial recognition device. However, as described above, the worker W can move freely. Therefore, the controller 30 will form a section 3000 for which calibration data has been acquired that is different from the target route information 850. In this case, the target route information 850 generated before the start becomes meaningless, so the controller 30 regenerates target route information 851 according to the section 3000 for which calibration data has been acquired and transmits it to the mobile communication terminal 800. As a result, the mobile communication terminal 800 can provide the worker W with the regenerated target route information 851.

[0125] The calibration status recognition unit 304 of the controller 30 may calculate the target route information 851 in real time when regenerating the target route information 851, or it may calculate the target route information 851 when it has moved a certain section or a certain amount of time. By calculating the target route information 851 when it has moved a certain section or a certain amount of time, it is possible to prevent the route from being changed sequentially.

[0126] For example, the calibration status recognition unit 304 may calculate the optimal target path information 851 based on the following requirements (a) to (c), etc.

[0127] (a) Starting from the current position of worker W, the shortest distance is prioritized such that the number of sections 3000 for which calibration data has been acquired is equal to or greater than the threshold.

[0128] (b) If an area that is difficult to walk through, such as an area where an obstacle has been detected, is recognized, that area is excluded.

[0129] (c) Generate multiple paths and evaluate which of the multiple paths is appropriate for a person to walk (avoiding sudden turns, U-turns, repeated back and forth, etc.) to optimize the path. For example, optimization algorithms include using Kruskal's method to calculate the shortest distance by considering the center of each section as a vertex.

[0130] In this way, the calibration system SYSγ can enable operator W to recognize a route in which calibration data can be efficiently obtained by regenerating target route information 851 based on the area moved by operator W during calibration (area 3000 where calibration data has been acquired).

[0131] Figure 14 shows other target path information 851 that is regenerated in the calibration system SYSγ. As described above, in the calibration of the spatial recognition device, it is not necessary to acquire calibration data for all sections 2001 in the calibration range 2000, but only to acquire calibration data for sections 2001 equal to the number of thresholds. For this reason, when generating target path information 851 for each section 2001 from which calibration data has not been acquired, target path information 851 that represents the shortest distance may be generated according to the number of sections 2001 equal to the number of thresholds.

[0132] For example, Figure 14 shows an example where it is sufficient to acquire calibration data for 14 out of 16 sections 2001 within the calibration range 2000. In this case, based on the above requirements (a) to (c), target route information 851 can be calculated that omits the acquisition of calibration data for two sections 2001. For example, the calibration status recognition unit 304 prioritizes areas with many sections 2001 for which calibration data has not been acquired, and calculates target route information 851 that is the shortest distance and results in an appropriate walking flow. This eliminates the need to move to areas with few sections 2001 for which calibration data has not been acquired, thereby reducing the burden on the worker W.

[0133] <Calibration of the spatial recognition device according to the fifth embodiment> Figure 15 is a flowchart showing the processing flow of the calibration method of the calibration system SYS according to the fifth embodiment. The calibration method of the calibration system SYS according to the fifth embodiment differs from the above calibration method in that a preliminary calibration is performed when a certain amount of calibration data has been collected for each section 2001, and a determination is made as to whether or not it is OK to terminate the acquisition of calibration data.

[0134] Here, while a certain amount of calibration data is generally required for calibration data collection, there is a possibility that the range of movement of the worker W may be limited by nearby cliffs, walls, obstacles, etc., making it impossible to collect calibration data for areas exceeding the threshold. Even if the calibration data is less than the threshold, the calibration process may still be able to be executed successfully. Therefore, the calibration system SYS includes a function to determine whether the calibration result calculated using calibration data collected at fewer locations than the threshold is correct. The determination of whether the calibration result is correct can be made mechanically by the controller 30, or by displaying the integrated result of a spatial recognition device using the collected calibration data via the notification unit for human visual confirmation. Thus, according to the fifth embodiment, it is possible to easily notify the worker that the calibration process is possible even if the collection of calibration data is incomplete.

[0135] The following describes the processing flow controlled by the controller 30 in the calibration method according to the fifth embodiment. Steps S121 to S123 in Figure 15 are the same as steps S101 to S103 in Figure 8, and a detailed explanation thereof will be omitted.

[0136] In step S124, the feature extraction unit 306 and calibration unit 307 of the controller 30 perform a preliminary calibration based on the calibration data of each section 2001 acquired as the worker W moves. In the preliminary calibration, the integrated result of the detection information from the imaging device S4, distance measuring device S5L, and S5R is confirmed using the calibration data collected up to the present. For example, in the preliminary calibration, calibration result image information is calculated and superimposed for each imaging device S4, distance measuring device S5L, and S5R based on the calibration data up to the present.

[0137] In the preliminary calibration, the controller 30 extracts the feature points of the calibration target (operator W) included in the calibration result image information of each imaging device S4 and distance measuring devices S5L and S5R for mechanical judgment (step S125). This allows the controller 30 to mechanically evaluate the misalignment between the superimposed feature points. For example, if the misalignment between feature points is large, it means that the calibration data has not been acquired sufficiently, and if the misalignment between feature points is small, it means that the calibration data has been acquired sufficiently.

[0138] Therefore, the controller 30 determines whether the misalignment between each feature point in each calibration result image information is less than the calibration threshold (step S126). If the misalignment between each feature point is greater than or equal to the calibration threshold (step S126: NO), it can be said that the acquisition of calibration data is insufficient, so the process returns to step S121 and repeats the same processing flow thereafter. On the other hand, if the misalignment between each feature point is less than the calibration threshold (step S126: YES), it can be said that the acquisition of calibration data is sufficient, and the process proceeds to step S127. If the combined result of the provisional calibration is to be judged by a person, it is advisable to configure the system so that the calibration result image information is displayed (notified) via the notification unit, and the person who checks this calibration result image information returns judgment information such as OK or NG.

[0139] The controller 30 then determines whether the number of sections 3000 for which calibration data has been acquired is greater than or equal to a certain number (step S127). That is, even if the deviation between each feature point is less than the calibration threshold in step S126, it is possible that the deviation between each feature point may be small by chance in a situation where there is little calibration data. For this reason, the controller 30 determines whether the number of sections 3000 for which calibration data has been acquired is greater than or equal to a certain number to prevent a shortage of calibration data. This "certain number" may be set to a number less than the number set as the threshold when acquiring calibration data for each section 2001 (for example, 14 in Figure 6).

[0140] If the number of sections 3000 for which calibration data has been acquired is less than a certain number (step S127: NO), the process returns to step S121 and repeats the same processing flow. On the other hand, if the number of sections 3000 for which calibration data has been acquired is greater than or equal to a certain number (step S127: YES), the process proceeds to step S128.

[0141] Step S128 is the same process as step S105 in Figure 8. That is, the output control unit 305 of the controller 30 notifies the worker W of the completion of work via the various components of the crane 100 (display device D1, speaker SP, light source L) or the portable communication terminal 800. By recognizing this completion of work information, the worker can recognize that the acquisition of calibration data has been completed. Alternatively, if the acquisition of calibration data is to be completed in a small number of sections, the notification unit (display device D1, speaker SP, light source L, portable communication terminal 800) may display (notify) the information of the integrated results that were combined in the preliminary calibration. Furthermore, the feature extraction unit 306 and calibration unit 307 of the controller 30 can use the calibration result information from the preliminary calibration to calibrate each spatial recognition device (imaging device S4, distance measuring devices S5L, S5R).

[0142] Thus, in the fifth embodiment, the controller 30 or the operator can proceed with the calibration work while confirming the calibration results. Therefore, the opportunities to redo the calibration work can be reduced as much as possible, making it possible to improve work efficiency and the reliability of the work.

[0143] <Note> The technical concept and effects of this disclosure, as described in the embodiments above, are described below.

[0144] A calibration system SYS for a spatial recognition device for a work machine according to a first aspect of this disclosure comprises: a plurality of spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) installed on a work machine (crane 100) and capable of acquiring the surrounding conditions of the work machine; an information processing device (controller 30) that performs calibration among the plurality of spatial recognition devices based on the position in which a predetermined object (worker W) present in the surrounding area of ​​the work machine is detected in the detection information of each of the plurality of spatial recognition devices; and a notification unit (display device D1, speaker SP, light source L, portable communication terminal 800) that notifies a person present in at least one of the surrounding area of ​​the work machine and / or inside the work machine, wherein the information processing device is configured to notify via the notification unit that the collection of calibration data has been completed.

[0145] As described above, the SYS calibration system for spatial recognition devices in industrial machinery can terminate the calibration work at an appropriate time by notifying the operator or related personnel that the acquisition of calibration data is complete. Therefore, even operators who are not experts can easily determine that suitable calibration data has been acquired. As a result, rework of calibration work can be prevented, and spatial recognition devices can be calibrated quickly and accurately. In other words, the SYS calibration system can improve the work efficiency when calibrating multiple spatial recognition devices.

[0146] Furthermore, the notification unit includes at least one of the following: a display device D1 provided on the work machine (crane 100), a sound output device (speaker SP), a light source L, or a portable communication terminal 800 that is connected to the information processing device (controller 30) in a manner that enables information communication. This allows the operator performing the calibration work to obtain information that the acquisition of calibration data has been completed by various means.

[0147] Furthermore, notification to a person includes at least one of the following: display of image information by display device D1, output of voice information or buzzer sound by sound output device (speaker SP), illumination of light source L, display of image information by portable communication terminal 800, output of voice information or buzzer sound by portable communication terminal 800, and vibration of portable communication terminal 800. This makes it possible for the operator performing the calibration work to more reliably recognize that the acquisition of calibration data has been completed.

[0148] Furthermore, calibration data is data from which the position coordinates of a predetermined object included in the detection information can be extracted. The information processing device (controller 30) monitors whether calibration data has been collected at multiple locations within the overlapping detection range of multiple spatial recognition devices, and if calibration data has been collected at multiple locations, it notifies that the collection of calibration data has been completed. This allows the information processing device to accurately determine the status of calibration data collection and to notify the completion of calibration data collection at an appropriate time.

[0149] Furthermore, the information processing device (controller 30) automatically starts the calibration process for multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) when calibration data has been collected at a preset number of locations. This allows the calibration system SYS to smoothly perform calibration of the spatial recognition devices without manual operation by an operator after the acquisition of calibration data is complete.

[0150] Furthermore, even if calibration data is collected at fewer locations than a preset number of locations, the information processing device (controller 30) performs calibration processing on multiple spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) and notifies the operator of the results of the calibration processing via the notification unit. This allows the operator to recognize the results of the calibration processing during the calibration data collection process and take appropriate action according to the results of the calibration processing.

[0151] Furthermore, the information processing device (controller 30) is configured to notify, via a notification unit, the degree to which the location of a predetermined object has been detected among multiple locations necessary for calibration when collecting calibration data. This allows the operator performing the calibration work to check the progress information while performing the calibration, further promoting the efficiency of the calibration work.

[0152] Furthermore, the information processing device (controller 30) notifies the operator via the notification unit of either the target route information 850 generated before the collection of calibration data, or the target route information 851 regenerated based on the calibration data during the calibration data collection process. This allows the operator to move while performing calibration work by referring to the target route information 850 and 851, enabling efficient collection of calibration data.

[0153] Furthermore, a second aspect of this disclosure is a work machine (crane 100) equipped with a plurality of spatial recognition devices (imaging device S4, distance measuring devices S5L, S5R) capable of acquiring the surrounding conditions, comprising: an information processing device (controller 30) that performs calibration among the plurality of spatial recognition devices based on the position in which a predetermined object present in the vicinity of the work machine is detected in the detection information of each of the plurality of spatial recognition devices; and a notification unit (imaging device S4, distance measuring devices S5L, S5R) that notifies a person present in at least one of the vicinity of the work machine and / or inside the work machine, wherein the information processing device is configured to notify via the notification unit that the collection of calibration data has been completed. Even in this case, the workability when performing calibration among the plurality of spatial recognition devices can be improved.

[0154] The calibration system SYS for a spatial recognition device for a work machine and the work machine according to the embodiments disclosed herein are illustrative and not restrictive in all respects. The embodiments can be modified and improved in various ways without departing from the scope and spirit of the appended claims. The matters described in the above embodiments can be otherwise configured and combined in a non-consistent manner.

[0155] This application claims priority to Japanese Patent Application No. 2024-230787, which was filed with the Japan Patent Office on December 26, 2024, and the entire contents of that application are incorporated herein by reference.

[0156] 30 Controller 100 Crane 800 Mobile communication terminal 850, 851 Target path information D1 Display device L Light source S4 Imaging device S5L, S5R Distancing device SP Speaker

Claims

1. A calibration system for spatial recognition devices for a work machine, comprising: a plurality of spatial recognition devices installed on the work machine and capable of acquiring the surrounding conditions of the work machine; an information processing device that performs calibration among the plurality of spatial recognition devices based on the location where a predetermined object present in the surrounding area of ​​the work machine is detected in the detection information of each of the plurality of spatial recognition devices; and a notification unit that notifies a person present in at least one of the surrounding area of ​​the work machine and / or inside the work machine, wherein the information processing device is configured to notify via the notification unit that the collection of calibration data has been completed.

2. The calibration system for a spatial recognition device for a work machine according to claim 1, wherein the notification unit includes at least one of a display device, a sound output device, a light source, or a portable communication terminal connected to the information processing device in a manner that enables information communication.

3. Calibration system for a spatial recognition device for a work machine according to claim 2, wherein the notification to the person includes at least one of the following: display of image information by the display device, output of voice information or a buzzer sound by the sound output device, illumination of the light source, display of image information by the mobile communication terminal, output of voice information or a buzzer sound by the mobile communication terminal, and vibration of the mobile communication terminal.

4. The calibration data is data from which the position coordinates of the predetermined object included in the detection information can be extracted, and the information processing device monitors whether the calibration data has been collected at multiple locations in the range where the detection ranges of the multiple spatial recognition devices overlap, and if the calibration data has been collected at multiple locations, it notifies that the collection of the calibration data has been completed, the calibration system for a spatial recognition device for a work machine according to any one of claims 1 to 3.

5. The information processing device automatically starts the calibration process for the plurality of spatial recognition devices when the calibration data has been collected at a predetermined number of the plurality of locations, the spatial recognition device calibration system for a work machine according to claim 4.

6. The information processing device performs calibration processing on the plurality of spatial recognition devices and notifies the result of the calibration processing via the notification unit, even when the calibration data is collected at a number of locations less than a preset number of locations, the spatial recognition device calibration system for a work machine according to claim 4.

7. The calibration system for a spatial recognition device for a work machine according to any one of claims 1 to 3, wherein the information processing device is configured to notify, via the notification unit, the degree to which the position of a predetermined object has been detected among a plurality of positions necessary for performing the calibration when collecting the calibration data.

8. The calibration system for a spatial recognition device for a work machine according to any one of claims 1 to 3, wherein the information processing device notifies via the notification unit of either target path information generated before the collection of the calibration data, or target path information regenerated based on the calibration data during the calibration data collection process.

9. A work machine equipped with a plurality of spatial recognition devices capable of acquiring the surrounding conditions, comprising: an information processing device that performs calibration among the plurality of spatial recognition devices based on the location where a predetermined object present in the vicinity of the work machine is detected in the detection information of each of the plurality of spatial recognition devices; and a notification unit that notifies a person present in at least one of the vicinity of the work machine and / or inside the work machine, wherein the information processing device is configured to notify via the notification unit that the collection of calibration data has been completed.

Images