System and method for monitoring surroundings of work machine, and work machine

Abstract

A system 10 for monitoring the surroundings of a hydraulic shovel 100 comprises cameras 11 for detecting the conditions around the hydraulic shovel 100, radars 13 for detecting the conditions around the hydraulic shovel 100 using a method different from that of the cameras 11, and a processor 1001. The processor 1001 acquires the speed of an obstacle with respect to the hydraulic shovel 100 on the basis of the detection data from the radars 13, determines whether the obstacle included in the detection data from the cameras 11 is a person on the basis of the detection data from the cameras 11 and the acquired speed of the obstacle with respect to the hydraulic shovel 100, and reports that a person has been detected when it is determined that the obstacle is a person.

Description

System, method for monitoring surroundings of work machine, and work machine

[0001] The present disclosure relates to a system, a method for monitoring surroundings of a work machine, and a work machine.

[0002] In the technical field related to work machines, there is known a technology as disclosed in Patent Document 1, which includes a plurality of cameras that acquire surrounding conditions of a work machine, and displays bird's-eye view images captured by the plurality of cameras and single-camera images. In the technique described in Patent Document 1, an obstacle is detected based on radar information, and when an obstacle is detected, a buzzer is activated to give a notification.

[0003] International Publication No. 2016 / 159012

[0004] When a work machine is traveling, it is desired to more accurately detect persons and the like existing around the work machine.

[0005] It is an object of the present disclosure to provide a system, a method for monitoring surroundings of a work machine, and a work machine, which can more accurately detect persons and the like existing around the work machine.

[0006] According to the present disclosure, there is provided a system for monitoring surroundings of a work machine, comprising: a first detection device for detecting surrounding conditions of the work machine; a second detection device for detecting surrounding conditions of the work machine by a method different from that of the first detection device; and a processor. The processor acquires a speed of an obstacle relative to the work machine based on detection data of the second detection device, determines whether the obstacle included in detection data of the first detection device is a person based on the detection data of the first detection device and the acquired speed of the obstacle relative to the work machine, and when it is determined that the obstacle is a person, notifies that a person has been detected.

[0007] This disclosure provides a method for monitoring the surroundings of a work machine, comprising: a first detection device for detecting the surrounding conditions of the work machine; a second detection device for detecting the surrounding conditions of the work machine in a manner different from that of the first detection device; and a controller having a processor. Based on the detection data of the second detection device, the speed of an obstacle relative to the work machine is obtained; based on the detection data of the first detection device and the obtained speed of the obstacle relative to the work machine, it is determined whether the obstacle included in the detection data of the first detection device is a person; and if it is determined that the obstacle is a person, a notification is given that a person has been detected.

[0008] The present disclosure provides a working machine comprising a slewing body, a traveling body supporting the slewing body, a first detection device attached to the slewing body for detecting the surrounding conditions of a shovel, a second detection device attached to the slewing body for detecting the surrounding conditions of a shovel in a manner different from that of the first detection device, and a controller having a processor. The controller acquires the speed of an obstacle relative to the shovel based on the detection data of the second detection device, determines whether the obstacle included in the detection data of the first detection device is a person based on the detection data of the first detection device and the acquired speed of the obstacle relative to the shovel, and if it determines that the obstacle is a person, it notifies that a person has been detected.

[0009] According to this disclosure, it is possible to more accurately detect people and other objects present around the work machine.

[0010] Figure 1 is a schematic diagram showing an example of a work machine. Figure 2 is a block diagram showing a system for monitoring the surroundings of a work machine according to the first embodiment. Figure 3 is a schematic diagram showing an example of an overhead view image and a single-camera image. Figure 4 is a diagram showing the criteria for determining a person or object. Figure 5 is a diagram showing an example of an overhead view image when a person is approaching the work machine. Figure 6 is a block diagram showing a computer system according to the first embodiment. Figure 7 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the first embodiment. Figure 8 is a block diagram showing a system for monitoring the surroundings of a work machine according to the second embodiment. Figure 9 is a diagram showing the relationship between the target travel speed and the relative speed threshold according to the second embodiment. Figure 10 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the second embodiment. Figure 11 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the third embodiment. Figure 12 is a block diagram showing a system for monitoring the surroundings of a work machine according to the fourth embodiment. Figure 13 is a diagram showing the direction of travel of the work machine and the detection area. Figure 14 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the fourth embodiment.

[0011] The embodiments described below will be explained with reference to the drawings, but the disclosure is not limited thereto. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0012] [First Embodiment] Figure 1 is a schematic diagram showing an example of a work machine. In this embodiment, the work machine 100 is a hydraulic excavator. In the following description, the work machine 100 will be referred to as a hydraulic excavator 100 as appropriate.

[0013] <Working Machine> The hydraulic excavator 100 comprises a hydraulically operated working machine 101, a slewing body 102 that supports the working machine 101, and a traveling body 103 that supports the slewing body 102. The slewing body 102 is capable of rotatable around the slewing axis RX while being supported by the traveling body 103.

[0014] The vehicle 103 has a pair of tracks 103C. The hydraulic excavator 100 moves as the tracks 103C rotate.

[0015] The work machine 101 has a boom 106 connected to a slewing body 102, an arm 107 connected to the tip of the boom 106, and a bucket 108 connected to the tip of the arm 107. The bucket 108 has a cutting edge 109.

[0016] The boom 106 is rotatable relative to the slewing body 102 around the boom axis AX1. The arm 107 is rotatable relative to the boom 106 around the arm axis AX2. The bucket 108 is rotatable relative to the arm 107 around the bucket axis AX3, the tilt axis AX4, and the rotate axis AX5, respectively. The boom axis AX1, the arm axis AX2, and the bucket axis AX3 are parallel to the Y axis. The tilt axis AX4 is perpendicular to the bucket axis AX3. The rotate axis AX5 is perpendicular to both the bucket axis AX3 and the tilt axis AX4. The slewing axis RX is parallel to the Z axis.

[0017] The X-axis direction is the front-to-back direction of the slewing body 102. The Y-axis direction is the vehicle width direction of the slewing body 102. The Z-axis direction is the up-and-down direction of the slewing body 102. With the slewing body 102 as a reference, the direction in which the work implement 101 is located is forward.

[0018] The work machine 101 is operated by power generated by a hydraulic cylinder 110. The hydraulic cylinder 110 is driven based on hydraulic fluid supplied from a hydraulic pump (not shown). The hydraulic cylinder 110 includes a boom cylinder 111, an arm cylinder 112, and a bucket cylinder 113. The boom cylinder 111 operates the boom 106. The boom cylinder 111 generates power to rotate the boom 106 around the boom axis AX1. The arm cylinder 112 operates the arm 107. The arm cylinder 112 generates power to rotate the arm 107 around the arm axis AX2. The bucket cylinder 113 operates the bucket 108. The bucket cylinder 113 generates power to rotate the bucket 108 around the bucket axis AX3.

[0019] <System for monitoring the surroundings of a work machine> Figure 2 is a block diagram showing a system for monitoring the surroundings of a hydraulic excavator according to the first embodiment. The system 10 includes a first detection device configured to detect the surrounding conditions of the hydraulic excavator 100, a second detection device configured to detect the surrounding conditions of the hydraulic excavator 100 in a different manner from the first detection device, a monitor 15 which is a display device, a speaker 17, a first controller 20, and a second controller 40. The second detection device detects the surrounding conditions of the hydraulic excavator 100 in a different manner from the first detection device. In this embodiment, the first detection device is a camera 11, and the second detection device is a radar 13.

[0020] Camera 11 consists of a group of cameras for capturing images of the surroundings of the hydraulic excavator 100. Camera 11 can capture images of obstacles located around the hydraulic excavator 100. Camera 11 is mounted on the slewing body 102 of the hydraulic excavator 100. The number of cameras 11 is not particularly limited. In this embodiment, camera 11 includes camera 11A, camera 11B, camera 11C, and camera 11D.

[0021] Camera 11A is configured to photograph the area in front of the hydraulic excavator 100. Camera 11A is positioned facing forward on the upper part of the rotating body 102 of the hydraulic excavator 100. Camera 11A outputs the captured image, which is the detection data, to the image processing unit 21 of the first controller 20.

[0022] Camera 11B is configured to photograph the right side of the hydraulic excavator 100. Camera 11B is positioned on the upper part of the rotating body 102 of the hydraulic excavator 100, facing to the right. Camera 11B outputs the captured image, which is the detection data, to the image processing unit 21 of the first controller 20.

[0023] Camera 11C is configured to photograph the left side of the hydraulic excavator 100. Camera 11C is positioned on the upper part of the rotating body 102 of the hydraulic excavator 100, facing to the left. Camera 11C outputs the captured image, which is the detection data, to the image processing unit 21 of the first controller 20.

[0024] Camera 11D is configured to photograph the rear of the hydraulic excavator 100. Camera 11D is positioned on the upper part of the rotating body 102 of the hydraulic excavator 100, facing rear. Camera 11D outputs the captured image, which is the detection data, to the image processing unit 21 of the first controller 20.

[0025] The radar 13 consists of a group of radars for detecting the surrounding conditions of the hydraulic excavator 100. The radar 13 can detect obstacles located around the hydraulic excavator 100. The radar 13 is mounted on the slewing body 102 of the hydraulic excavator 100. The radar 13 may be configured to have a shorter detection time compared to, for example, the camera 11. The number of radars 13 is not particularly limited. In this embodiment, the radar 13 includes radar 13A, radar 13B, radar 13C, and radar 13D.

[0026] The radar 13A is configured to detect obstacles to the right front of the hydraulic excavator 100. The radar 13A is positioned at the lower part of the rotating body 102 of the hydraulic excavator 100, facing forward to the right. The radar 13A outputs detection data of detected obstacles to the obstacle processing unit 26 of the first controller 20.

[0027] Radar 13B is configured to detect an obstacle to the right rear of the hydraulic excavator 100. Radar 13B is positioned below the rotating body 102 of the hydraulic excavator 100, facing to the right rear. Radar 13B outputs detection data of detected obstacles to the obstacle processing unit 26 of the first controller 20.

[0028] The radar 13C is configured to detect obstacles behind the hydraulic excavator 100. The radar 13C is positioned facing rearward at the bottom of the slewing body 102 of the hydraulic excavator 100. The radar 13C outputs detection data of detected obstacles to the obstacle processing unit 26 of the first controller 20.

[0029] The radar 13D is configured to detect an obstacle to the left rear of the hydraulic excavator 100. The radar 13D is positioned facing left rear at the bottom of the rotating body 102 of the hydraulic excavator 100. The radar 13D outputs detection data of the detected obstacle to the obstacle processing unit 26 of the first controller 20.

[0030] The monitor 15 is configured to receive display signals output from the first controller 20. The monitor 15 displays, for example, an overhead view image 200 (see Figure 3). The monitor 15 is located, for example, in the operator's cab of the hydraulic excavator 100.

[0031] <First Controller> The first controller 20 includes a numerical processing unit (processor) such as a CPU. The first controller 20 is located on the hydraulic excavator 100. The first controller 20 acquires captured images, which are detection data, from the camera 11 and acquires detection data indicating the detection of obstacles from the radar 13. Based on the detection data acquired from the camera 11 and the radar 13, the first controller 20 detects people or objects. The first controller 20 is configured to notify the operator of the work machine 100 that people or objects have been detected around the hydraulic excavator 100. The first controller 20 is configured to output a command signal to the second controller 40 when people or objects are detected around the hydraulic excavator 100. The first controller 20 determines that the obstacle in the image captured by the camera 11 is a person if the value indicating human-likeness calculated based on the image captured by the camera 11 is greater than or equal to a first threshold and less than a second threshold greater than the first threshold, and if it determines, based on detection data from the camera 11 or radar 13, that an obstacle is approaching the hydraulic excavator 100 at a predetermined speed or higher. The first controller 20 may determine that the obstacle in the image captured by the camera 11 is a person if the value indicating human-likeness is greater than or equal to a third threshold greater than the second threshold, regardless of detection data from radar 13. The first controller 20 may also determine that the obstacle in the image captured by the camera 11 is a person if the value indicating human-likeness is greater than or equal to a second threshold and less than the third threshold, and if the coordinates of the obstacle recognized based on the detection data from the camera 11 and the coordinates of the obstacle recognized based on the detection data from radar 13 are within a predetermined range. As an example of implementing the above functions, the first controller 20 includes an image processing unit 21, an obstacle processing unit 26, a speed acquisition unit 28, and a display control unit 29.

[0032] Figure 6 is a block diagram illustrating a computer system according to the first embodiment. The first controller 20 and the second controller 40, described later, comprise a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the first controller 20 and the second controller 40 are stored as programs in the storage 1003. The processor 1001 reads the programs from the storage 1003, loads them into the main memory 1002, and executes the above-described processes according to the programs. The programs may be distributed to the computer system 1000 via a network.

[0033] The image processing unit 21 acquires images from the camera 11 and performs image processing. In this embodiment, the overhead image generation unit 22 acquires four images from cameras 11A, 11B, 11C, and 11D and generates an overhead image 200. The image processing unit 21 comprises the overhead image generation unit 22, the image synthesis unit 23, the first recognition unit 24, and the determination unit 25.

[0034] Figure 3 is a schematic diagram showing an example of an overhead view image and a single-camera image. The overhead view image generation unit 22 generates an overhead view image 200 based on multiple images acquired from the camera 11. In this embodiment, the overhead view image generation unit 22 converts four images acquired from cameras 11A, 11B, 11C, and 11D into an image viewed from above. The overhead view image generation unit 22 converts the image into an image viewed from a predetermined virtual viewpoint located above the hydraulic excavator 100. More specifically, the overhead view image generation unit 22 performs an image conversion to project from the virtual viewpoint above the hydraulic excavator 100 onto a predetermined virtual projection plane corresponding to the ground surface level. Subsequently, the overhead view image generation unit 22 extracts the converted images corresponding to each area of ​​the frame displaying the overhead view image 200 and synthesizes each converted image within the frame. Icon images 210 corresponding to the plan view of the hydraulic excavator 100 are placed in the overhead view image 200 generated by the overhead view image generation unit 22. The method for generating the overhead view image 200 is not limited to this, and known methods can be used.

[0035] The image synthesis unit 23 synthesizes an image onto the overhead view image 200. When obstacle information is input from the obstacle processing unit 26 (described later), the image synthesis unit 23 generates an image in which the obstacle information is synthesized onto the overhead view image 200.

[0036] The image synthesis unit 23 generates an image by compositing guide lines 211 onto the overhead view image 200. The image synthesis unit 23 composites guide lines 211 on the overhead view image 200 that indicate the range for controlling at least one of driving and turning.

[0037] The guideline 211 indicates the range within which at least one of the travel and rotation of the hydraulic excavator 100 is controlled when an obstacle is detected around the hydraulic excavator 100. The guideline 211 is an image surrounding the hydraulic excavator 100. In this embodiment, the guideline 211 consists of a first guideline 213 and a second guideline 215.

[0038] The first guideline 213 is the range (stopping area) within which the hydraulic excavator 100 is controlled to stop moving and rotating when an obstacle is detected around the hydraulic excavator 100. On the side of the hydraulic excavator 100's body, the first guideline 213 indicates the stopping distance from the center of rotation of the body. In the example shown in Figure 3, the first guideline 213 is a straight line on the side of the hydraulic excavator 100's body. On the rear of the hydraulic excavator 100's body, the first guideline 213 indicates the stopping distance from the rear end of the body. In the example shown in Figure 3, the first guideline 213 is an arc on the rear of the hydraulic excavator 100's body.

[0039] The second guideline 215 is a range (deceleration region) in which the movement of the hydraulic excavator 100 is controlled to slow down when an obstacle is detected around the hydraulic excavator 100. On the side of the hydraulic excavator 100's body, the second guideline 215 is a straight line indicating a predetermined distance outward from the first guideline 213 on the side of the body. On the rear of the hydraulic excavator 100's body, it is a straight line passing through a predetermined distance outward from the position of the first guideline 213 at the rear of the body that is furthest from the center of rotation.

[0040] The image synthesis unit 23 is configured to generate an image that visually notifies the operator of the detection of a person or object when the determination unit 25, described later, determines that a person or object has been detected. The image synthesis unit 23 may overlay an image indicating the location where the person or object was detected onto the overhead image 200 based on obstacle information acquired from the determination unit 25, described later. The image synthesis unit 23 may provide different notification methods depending on whether a person or an object is detected. When the determination unit 25, described later, determines that a person has been detected, the image synthesis unit 23 may, based on obstacle information, for example, synthesize a marker image indicating that a person or object has been detected onto the overhead image 200. When the determination unit 25, described later, determines that an object has been detected, the image synthesis unit 23 may, based on obstacle information, for example, synthesize a marker image indicating that an object has been detected onto the overhead image 200. If the determination unit 25 (described later) determines that a person has been detected, the image synthesis unit 23 may synthesize, for example, an indicator or pop-up indicating that a person has been detected, based on the obstacle information. If the determination unit 25 (described later) determines that an object has been detected, the image synthesis unit 23 may synthesize, for example, an indicator or pop-up indicating that an object has been detected, based on the obstacle information.

[0041] The first recognition unit 24 recognizes a person from the image captured by the camera 11. In this embodiment, the first recognition unit 24 recognizes a person from the images captured by cameras 11A, 11B, 11C, and 11D using a person recognition dictionary. The person recognition dictionary is, for example, a dictionary of feature quantities extracted from each of several known images in which a person is depicted. Examples of feature quantities that can be used include HOG (Histograms of Oriented Gradients) and CoHOG (Co-occurrence HOG). A known method can be used to recognize a person from an image.

[0042] In the following explanation, the first threshold will be referred to as threshold C, the second threshold as threshold B, and the third threshold as threshold A. The relationship between thresholds A, B, and C is that threshold C < threshold B < threshold A.

[0043] In the embodiment, the first recognition unit 24 uses a human recognition dictionary from an image captured by the camera 11 to calculate a value indicating human-likeness (hereinafter referred to as "score P") to recognize a person. A larger score P indicates a higher possibility that the obstacle included in the image captured by the camera 11 is a human. For example, when the score P is equal to or greater than a threshold A, the first recognition unit 24 recognizes a "person" from the image captured by the camera 11. For example, when the score P is smaller than the threshold A and equal to or greater than a threshold B which is smaller than the threshold A, the first recognition unit 24 recognizes a "human-like" obstacle from the image captured by the camera 11. The first recognition unit 24 outputs the recognition result to the determination unit 25.

[0044] The recognition result of the first recognition unit 24 includes coordinates indicating the position where a "person" or a "human-like" obstacle is recognized.

[0045] The determination unit 25 will be described after describing the obstacle processing unit 26 and the speed acquisition unit 28.

[0046] The obstacle processing unit 26 detects humans and objects that are obstacles from detection data of the radar 13. The obstacle processing unit 26 includes a second recognition unit 27.

[0047] The second recognition unit 27 detects humans and objects that are obstacles from detection data of the radar 13. In the embodiment, the second recognition unit 27 detects humans and objects that are obstacles from detection data of the radar 13A, the radar 13B, the radar 13C and the radar 13D. A publicly known method can be used as the method for detecting humans and objects that are obstacles from detection data of the radar 13. The second recognition unit 27 outputs the detection result to the determination unit 25 of the image processing unit 21.

[0048] The recognition result of the second recognition unit 27 includes coordinates indicating the position where a human or an object that is an obstacle is detected.

[0049] Based on detection data of the radar 13, for example, if the height, width or aspect ratio of the outer shape of the recognized obstacle is out of an appropriate range for the height, width or aspect ratio of a human, the second recognition unit 27 may detect the obstacle as an object instead of detecting it as a human.

[0050] The speed acquisition unit 28 acquires the speed of the obstacle relative to the hydraulic excavator 100, in other words, the relative speed between the hydraulic excavator 100 and the obstacle. The speed acquisition unit 28 acquires the speed of the obstacle relative to the hydraulic excavator 100 from the recognition result of the first recognition unit 24 or the recognition result of the second recognition unit 27. In the embodiment, the speed acquisition unit 28 acquires the speed of the obstacle relative to the hydraulic excavator 100 from the detection data of the radar 13. For example, the speed acquisition unit 28 acquires the speed of the obstacle relative to the hydraulic excavator 100 from the position of the obstacle recognized by the second recognition unit 27.

[0051] The determination unit 25 determines whether the obstacle is a person or not based on the recognition result of the first recognition unit 24, the recognition result of the second recognition unit 27, and the speed of the obstacle relative to the hydraulic excavator 100 acquired by the speed acquisition unit 28. In this embodiment, the determination unit 25 determines whether the obstacle is a person or an object based on the recognition result of the first recognition unit 24, the recognition result of the second recognition unit 27, and the speed of the obstacle relative to the hydraulic excavator 100 acquired by the speed acquisition unit 28.

[0052] The criteria for determining whether something is a person or an object in the determination unit 25 will be explained using Figure 4. Figure 4 is a diagram showing the criteria for determining whether something is a person or an object.

[0053] The determination unit 25 determines that an obstacle included in the image captured by the camera 11 is a person if any of the following determination conditions (a1) to (a3) ​​are met: (a1) The score P calculated by the first recognition unit 24 is greater than or equal to threshold A. (a2) The score P calculated by the first recognition unit 24 is less than threshold A and greater than or equal to threshold B, and the second recognition unit 27 has detected an obstacle. (a3) ​​The score P calculated by the first recognition unit 24 is less than threshold B and greater than or equal to threshold C, and the second recognition unit 27 has detected an obstacle, and the obstacle is approaching the hydraulic excavator 100 at a predetermined speed or greater.

[0054] (a1) Regardless of whether the second recognition unit 27 detects an obstacle, if the score P is equal to or greater than the threshold A, it is determined that the obstacle included in the image captured by the camera 11 is a person.

[0055] The predetermined speed in (a3), in other words, the relative speed threshold, is, for example, 1.2 [m / s]. The relative speed threshold is the speed threshold of the obstacle relative to the hydraulic excavator 100.

[0056] The phrase (a3) ​​that an obstacle is approaching the hydraulic excavator 100 at a predetermined speed or higher does not mean that the obstacle is moving toward the hydraulic excavator 100 at a predetermined speed or higher. For example, it may also include cases where the hydraulic excavator 100 is moving toward the obstacle while the obstacle is stationary or moving at a speed below the predetermined speed. In other words, the phrase that an obstacle is approaching the hydraulic excavator 100 at a predetermined speed or higher means that the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher. Hereinafter, the predetermined speed will be referred to as the relative speed threshold.

[0057] (a3) The case in which an obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher may also include the case in which the obstacle is approaching the slewing body 102 of the hydraulic excavator 100 at a predetermined speed or higher when the slewing body 102 of the hydraulic excavator 100 is rotating.

[0058] Furthermore, if the hydraulic excavator 100 is traveling or turning in a direction away from the obstacle, and the obstacle is moving toward the hydraulic excavator 100, this does not need to be included in the case where the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher in (a3).

[0059] The determination unit 25 determines that an obstacle included in the image captured by the camera 11 is an object if either of the following determination conditions (b1) or (b2) is met: (b1) The score P calculated by the first recognition unit 24 is less than threshold B and greater than or equal to threshold C, and the second recognition unit 27 has detected an obstacle, and the obstacle is not approaching the hydraulic excavator 100 at a relative speed threshold or greater. (b2) The score P calculated by the first recognition unit 24 is less than threshold C, and the second recognition unit 27 has detected an obstacle.

[0060] The determination unit 25 does not detect people and objects if the following determination condition (c1) is met: (c1) The score P calculated by the first recognition unit 24 is less than the threshold C, and the second recognition unit 27 has not detected an obstacle.

[0061] Figure 5 shows an example of an overhead view image of a situation where a person 300 is approaching a hydraulic excavator 100. In Figure 5, the person 300 is approaching the hydraulic excavator 100 at a relative speed threshold or higher. When the radar 13 has detected an obstacle and the score P is less than threshold B and greater than or equal to threshold C, and the obstacle (person 300) is approaching the hydraulic excavator 100 at a relative speed threshold or higher, as shown in Figure 5, the determination unit 25 determines that the obstacle included in the image captured by the camera 11 is a person. In this case, a marker image 300M indicating that a person 300 has been detected is superimposed on the overhead view image 200.

[0062] The determination unit 25 may determine that the obstacle included in the image captured by the camera 11 is a person if the score P calculated by the first recognition unit 24 is less than threshold A and greater than or equal to threshold B, and the second recognition unit 27 has detected an obstacle, and the coordinates of the obstacle recognized by the first recognition unit 24 and the coordinates of the obstacle detected by the second recognition unit 27 are within a predetermined range. The predetermined range is, for example, a radius of about 2m.

[0063] The determination unit 25 may determine that the obstacle included in the image captured by the camera 11 is a person if the score P calculated by the first recognition unit 24 is less than threshold B and greater than or equal to threshold C, and the second recognition unit 27 has detected an obstacle, and the coordinates of the obstacle recognized by the first recognition unit 24 and the coordinates of the obstacle detected by the second recognition unit 27 are within a predetermined range, and the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or greater. The predetermined range is, for example, a radius of about 2m.

[0064] After the determination unit 25 determines that an obstacle included in the image captured by the camera 11 is a person, it may determine that a person has been detected if the second recognition unit 27 detects an obstacle within a predetermined distance range. The predetermined distance range is, for example, a radius of about 2 meters. The predetermined distance range is, for example, the distance a person can travel in a very short time, such as about 1 second.

[0065] The determination unit 25 may determine that the obstacle included in the image captured by the camera 11 is an object if the score P calculated by the first recognition unit 24 is less than threshold A and greater than or equal to threshold C, and the second recognition unit 27 has detected an obstacle, and the coordinates of the obstacle recognized by the first recognition unit 24 and the coordinates of the obstacle detected by the second recognition unit 27 are not within a predetermined range.

[0066] The determination unit 25 may determine that the obstacle included in the image captured by the camera 11 is an object if the score P calculated by the first recognition unit 24 is less than threshold B and greater than or equal to threshold C, and the second recognition unit 27 has detected an obstacle, and the coordinates of the obstacle recognized by the first recognition unit 24 and the coordinates of the obstacle detected by the second recognition unit 27 are within a predetermined range, and the obstacle is not approaching the hydraulic excavator 100 at a relative speed threshold or higher.

[0067] If the determination unit 25 determines that it has detected a person or an object, it may sound the speaker 17 to notify the operator. The determination unit 25 may provide different notification methods depending on whether it has detected a person or an object. The determination unit 25 may sound the speaker 17 to audibly notify the operator that the detected obstacle is a person. The determination unit 25 may sound the speaker 17 to audibly notify the operator that the detected obstacle is an object. Examples of notification methods include voice, volume, pitch, etc. If the determination unit 25 determines that it has detected a person or an object, it may output obstacle information indicating the size and position (coordinates) of the detected person or object to the image synthesis unit 23.

[0068] After the determination unit 25 determines that a person has been detected, it may maintain the determination result that a person has been detected for a predetermined period of time.

[0069] The display control unit 29 controls the display of various images on the monitor 15. The display control unit 29 generates a display signal to display the overhead image 200 input from the image synthesis unit 23 on the monitor 15.

[0070] The display control unit 29 acquires multiple images from the camera 11 and generates a single-camera image. The multiple images may be acquired from the image processing unit 21. The display control unit 29 generates a display signal to display, for example, a single-camera image 220 of the rear of the hydraulic excavator 100, taken by the camera 11D, on the monitor 15. As shown in Figure 3, the rear single-camera image 220 may display guide lines 221 indicating a predetermined distance from the rear of the hydraulic excavator 100. In the rear single-camera image 220, three guide lines 221A, 221B, and 221C are displayed, but the number of guide lines 221 displayed is not limited to three; there may be two or fewer, or four or more.

[0071] The guideline 221 of the rear single-camera image 220 may be displayed at positions corresponding to the first guideline 213 and the second guideline 215 in the overhead view image 200. In this case, the operator can visually confirm, not only in the overhead view image 200 but also in the rear single-camera image 220, where the range for controlling at least one of the travel and rotation of the hydraulic excavator 100 is located.

[0072] The display control unit 29 may, instead of the image synthesis unit 23, generate a display signal to overlay an image indicating the location where a person or object was detected onto the overhead image 200, based on obstacle information acquired from the determination unit 25. If the determination unit 25 determines that a person or object has been detected, the display control unit 29 may generate a display signal to display, for example, a marker image indicating that a person or object has been detected onto the overhead image 200. If the determination unit 25 determines that a person or object has been detected, the display control unit 29 may generate a display signal to display, for example, an indicator or pop-up indicating that a person or object has been detected.

[0073] <Second Controller> The second controller 40 includes a numerical processing unit (processor) such as a CPU. The second controller 40 is located on the hydraulic excavator 100. The second controller 40 is configured to control the operation of the hydraulic excavator 100 based on command signals from the first controller. The second controller 40 is configured to control the operation of the hydraulic excavator 100 when the first controller 20 determines that a person or object has been detected around the hydraulic excavator 100. The second controller 40 includes a work machine control unit 41.

[0074] The work machine control unit 41 outputs various control signals to control the hydraulic excavator 100. When the determination unit 25 of the first controller 20 determines that a person or object has been detected, the work machine control unit 41 controls at least one of the travel and rotation of the hydraulic excavator 100. When the determination unit 25 determines that a person or object has been detected within the stopping area, the work machine control unit 41 controls the travel and rotation of the hydraulic excavator 100 to stop. When the determination unit 25 determines that a person or object has been detected within the deceleration area, the work machine control unit 41 controls the travel and rotation of the hydraulic excavator 100 to slow down.

[0075] The work machine control unit 41 controls the hydraulic excavator 100 by, for example, changing the travel speed limit from "high" to "low", changing the travel speed limit from "medium" to "low", stopping travel, stopping slewing, or stopping the work machine 101.

[0076] <An example of a method for monitoring the surroundings of a work machine> Figure 7 is an example of a flowchart showing a method for monitoring the surroundings of a hydraulic excavator according to the first embodiment. When the hydraulic excavator 100 is started, the system 10 for monitoring the surroundings of the hydraulic excavator 100 is activated. When the system 10 is activated, the processing shown in the flowchart in Figure 7 is started. During the execution of the processing shown in the flowchart in Figure 7, the identification processing by the first recognition unit 24 and the second recognition unit 27 is continuously performed.

[0077] The first controller 20 acquires an image from the camera 11 using the first recognition unit 24 (step ST11). The first controller 20 uses the image captured by the camera 11 and pre-stored human recognition dictionary data to calculate a score P, which is a value indicating human-likeness. The first controller 20 proceeds to step ST12.

[0078] The first controller 20 determines, using the determination unit 25, whether the human-likeness is above the threshold C (step ST12). More specifically, the first controller 20 determines, using the determination unit 25, whether the score P, which is a value indicating human-likeness calculated by the first recognition unit 24, is above the threshold C. If the first controller 20 determines, using the determination unit 25, that the score P, which is a value indicating human-likeness, is above the threshold C (step ST12; Yes), it proceeds to step ST13. If the first controller 20 determines, using the determination unit 25, that the score P, which is a value indicating human-likeness, is not above the threshold C (step ST12; No), it proceeds to step ST22.

[0079] If the first controller 20 determines that the human-likeness is above threshold C (step ST12; Yes), the first controller 20 uses the determination unit 25 to determine whether or not the human-likeness is above threshold A (step ST13). More specifically, the first controller 20 uses the determination unit 25 to determine whether or not the recognition result of the first recognition unit 24 indicates that the score P, which is a value indicating human-likeness, is above threshold A. If the first controller 20 determines, using the determination unit 25, that the score P, which is a value indicating human-likeness, is above threshold A (step ST13; Yes), the process proceeds to step ST19. If the first controller 20 determines, using the determination unit 25, that the score P, which is a value indicating human-likeness, is not above threshold A (step ST13; No), the process proceeds to step ST14.

[0080] If the first controller 20 determines that the human-likeness is not above threshold A (step ST13; No), the first controller 20 determines, using the determination unit 25, whether or not the radar 13 has detected an obstacle (step ST14). More specifically, the first controller 20 determines, using the determination unit 25, whether or not the second recognition unit 27 has detected an obstacle. If the first controller 20 determines, using the determination unit 25, that the radar 13 has detected an obstacle (step ST14; Yes), it proceeds to step ST15. If the first controller 20 determines, using the determination unit 25, that the radar 13 has not detected an obstacle (step ST14; No), it terminates the processing of this flowchart.

[0081] If the radar 13 determines that an obstacle has been detected (step ST14; Yes), the first controller 20 uses the determination unit 25 to determine whether the coordinates of the obstacle are within a predetermined range (step ST15). More specifically, the first controller 20 uses the determination unit 25 to determine whether the coordinates where a person was recognized from the image of the camera 11 and the coordinates where the obstacle was detected from the detection data of the radar 13 are within a predetermined range. If the first controller 20 determines that the coordinates of the obstacle are within a predetermined range (step ST15; Yes), the process proceeds to step ST16. If the first controller 20 determines that the coordinates of the obstacle are not within a predetermined range (step ST15; No), the process proceeds to step ST18.

[0082] If the first controller 20 determines that the coordinates of the obstacle are within a predetermined range (step ST15; Yes), the first controller 20 uses the determination unit 25 to determine whether or not the human-likeness is above threshold B (step ST16). More specifically, the first controller 20 uses the determination unit 25 to determine whether or not the recognition result of the first recognition unit 24 indicates that the score P, which is a value indicating human-likeness, is above threshold B. If the first controller 20 determines that the score P, which is a value indicating human-likeness, is above threshold B (step ST16; Yes), the process proceeds to step ST19. If the first controller 20 determines that the score P, which is a value indicating human-likeness, is not above threshold B (step ST16; No), the process proceeds to step ST17.

[0083] If the first controller 20 determines that the score P, which is a value indicating human-likeness, is not equal to or greater than the threshold B (step ST16; No), the first controller 20 uses the determination unit 25 to determine whether or not the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher (step ST17). More specifically, the first controller 20 uses the determination unit 25 to determine, based on the recognition result of the second recognition unit 27, whether or not the recognized obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher. If the first controller 20 determines, using the determination unit 25, that the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher (step ST17; Yes), the process proceeds to step ST19. If the first controller 20 determines, using the determination unit 25, that the obstacle is not approaching the hydraulic excavator 100 at a relative speed threshold or higher (step ST17; No), the process proceeds to step ST18.

[0084] The first controller 20 determines, using the determination unit 25, that the obstacle included in the image captured by the camera 11 is an object (step ST18). The first controller 20 proceeds to step ST20.

[0085] The first controller 20 determines, using the determination unit 25, that the obstacle included in the image captured by the camera 11 is a person (step ST19). The first controller 20 proceeds to step ST20.

[0086] The first controller 20 notifies the operator that a person or object has been detected based on the determination result made in step ST18 or step ST19 (step ST20). The first controller 20 then proceeds to step ST21.

[0087] The first controller 20 controls the operation of the hydraulic excavator 100 based on the determination result determined in step ST18 or step ST19 (step ST21).

[0088] If the first controller 20 determines that the score P, which is a value indicating human-likeness, is not equal to or greater than the threshold C (step ST12; No), the first controller 20 uses the determination unit 25 to determine whether or not the radar 13 has detected an obstacle (step ST22). If the first controller 20 determines, using the determination unit 25, that the radar 13 has detected an obstacle (step ST22; Yes), the process proceeds to step ST18. If the first controller 20 determines, using the determination unit 25, that the radar 13 has not detected an obstacle (step ST22; No), the process of this flowchart ends.

[0089] <Effects> As described above, in the embodiment, regardless of whether or not the radar 13 detects an obstacle, if the score P, which is a value indicating human-likeness, is greater than or equal to threshold A, it can be determined that the obstacle included in the image captured by the camera 11 is a person. In the embodiment, if the radar 13 has detected an obstacle, and the score P is less than threshold A and greater than or equal to threshold B, and the radar 13 has detected an obstacle based on the recognition result from the detection data it has detected, it can be determined that the obstacle included in the image captured by the camera 11 is a person. In the embodiment, if the radar 13 has detected an obstacle, and the score P is less than threshold B and greater than or equal to threshold C, and the obstacle is approaching the hydraulic excavator 100 at a relative speed threshold or higher, it can be determined that the obstacle included in the image captured by the camera 11 is a person. According to the embodiment, even if the score P, which is a value indicating human-likeness, is small, if the obstacle is relatively close to the hydraulic excavator 100, it can be determined that the obstacle included in the image captured by the camera 11 is a person. According to the embodiment, even when the hydraulic excavator 100 is moving, people can be detected more appropriately. According to this embodiment, it is possible to more accurately detect people and other objects present around the hydraulic excavator 100.

[0090] In this embodiment, if it is determined that a person has been detected, a marker image indicating that a person has been detected can be displayed. According to this embodiment, the operator can be appropriately notified that the detected obstacle is a person.

[0091] In this embodiment, if it is determined that a person has been detected, an indicator or pop-up can be displayed to show that a person has been detected. According to this embodiment, the operator can be appropriately notified that the detected obstacle is a person.

[0092] In this embodiment, if it is determined that a person has been detected, the speaker 17 can be made to sound to audibly inform the operator that the detected obstacle is a person. According to this embodiment, the operator can be appropriately informed that the detected obstacle is a person.

[0093] In this embodiment, the hydraulic excavator 100 can be controlled when it is determined that a person or object has been detected. In this embodiment, the hydraulic excavator 100 can be controlled by changing the travel speed limit from "high" to "low", changing the travel speed limit from "medium" to "low", stopping travel, or stopping the slewing.

[0094] In the embodiments described above, the work machine is not limited to a hydraulic excavator. The work machine can be a dump truck, a wheel loader, or other work machine.

[0095] [Second Embodiment] Figure 8 is a block diagram showing a system for monitoring the surroundings of a work machine according to the second embodiment. Figure 9 is a diagram showing the relationship between the target travel speed and the relative speed threshold according to the second embodiment. Figure 10 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the second embodiment. In this embodiment, the setting of the relative speed threshold differs from that of the first embodiment. Other aspects are configured in the same way as in the first embodiment. The same configuration as in the first embodiment will not be described.

[0096] The hydraulic excavator 100 is equipped with an operating device 120. The operating device 120 includes, for example, a mode setting unit 121, an output adjustment unit 122, a work machine operation unit 123, and an enable / disable switching unit 124. The operating device 120 is configured to be operable by an operator. The operating device 120 is located inside the operator's cab. The operating device 120 is communicated with the first controller 20.

[0097] The mode setting unit 121 is an operating device for setting the travel mode of the hydraulic excavator 100. The mode setting unit 121 is configured to allow an operator to set one of the following travel modes: "high," "middle," or "low." When the travel mode is set to "high," the travel speed limit of the hydraulic excavator 100 can be set higher than when the travel mode is set to "middle." When the travel mode is set to "middle," the travel speed limit of the hydraulic excavator 100 can be set higher than when the travel mode is set to "low." The mode setting unit 121 may be displayed on the display screen of the monitor 15, for example. The mode setting unit 121 transmits a control command signal indicating the set value of the travel mode to the first controller 20.

[0098] The output adjustment unit 122 is an operating device for adjusting the output of the power source of the hydraulic excavator 100. The power source of the hydraulic excavator 100 may be an engine or an electric motor. The output adjustment unit 122 is configured, for example, to allow an operator to adjust the rotational speed of the power source. The output adjustment unit 122 may be, for example, a dial that can be operated by an operator, or it may be displayed on the display screen of the monitor 15. The output adjustment unit 122 transmits a control command signal indicating the target output of the power source to the first controller 20.

[0099] The work machine operation unit 123 is an operating device for controlling the hydraulic excavator 100. The work machine operation unit 123 is configured to allow, for example, an operator to control the hydraulic excavator 100. The work machine operation unit 123 includes, for example, a work machine operation lever 123A, a slewing operation lever 123B, and a travel operation lever 123C. The work machine operation unit 123 transmits a control command signal indicating the amount of operation to the first controller 20.

[0100] The implement control lever 123A is an operating device for controlling the operation of the implement 101. The implement control lever 123A is configured to allow the operator to control the operation of the boom 106, arm 107, and bucket 108, respectively. The implement control lever 123A transmits a control command signal indicating the amount of operation to the first controller 20.

[0101] The slewing control lever 123B is an operating device for controlling the slewing motion of the slewing body 102 relative to the traveling body 103. The slewing control lever 123B is configured to allow an operator to control the slewing motion of the slewing body 102 relative to the traveling body 103. The work equipment control lever 123A transmits a control command signal indicating the amount of operation to the first controller 20.

[0102] The travel control lever 123C comprises a left travel control lever 123CL and a right travel control lever 123CR, which are operated to move the vehicle 103. When the left travel control lever 123CL is operated in the forward / backward direction, the left track 103C of the vehicle 103 moves forward or backward. When the right travel control lever 123CR is operated in the forward / backward direction, the right track 103C of the vehicle 103 moves forward or backward. When it is not necessary to distinguish between the left travel control lever 123CL and the right travel control lever 123CR, they will be described as the travel control lever 123C. The travel control lever 123C transmits a control command signal indicating the amount of operation to the first controller 20.

[0103] The enable / disable switch 124 is an operating device for enabling or disabling the operation of the hydraulic excavator 100. The enable / disable switch 124 is configured to enable or disable the operation of the hydraulic excavator 100 by an operator, for example. When the enable / disable switch 124 is operated by the operator to enable the operation of the hydraulic excavator 100, the work implement 101, the slewing body 102, and the traveling body 103 become operational in response to the operation of the work machine operation unit 123. When the enable / disable switch 124 is operated by the operator to disable the operation of the hydraulic excavator 100, the work implement 101, the slewing body 102, and the traveling body 103 become inoperable regardless of the operation of the work machine operation unit 123. The enable / disable switch 124 transmits at least one of the control command signals for enabling the operation of the hydraulic excavator 100 and the control command signal for disabling the operation of the hydraulic excavator 100 to the first controller 20.

[0104] The determination unit 25 sets a relative speed threshold used for determining whether a person or object is present, according to the travel speed of the hydraulic excavator 100. In this embodiment, the relative speed threshold used for determining whether a person or object is present is described as being set according to the target travel speed of the hydraulic excavator 100.

[0105] The target travel speed is determined according to the amount of operation of the output adjustment unit 122, the amount of operation of the travel operation lever 123C, and the setting value of the travel mode. The larger the amount of operation of the travel operation lever 123C, the faster the target travel speed of the hydraulic excavator 100. The larger the amount of operation of the output adjustment unit 122, the faster the target travel speed of the hydraulic excavator 100 can be set. When the travel mode is set to "high," the target travel speed of the hydraulic excavator 100 can be set faster than when the travel mode is set to "middle." When the travel mode is set to "middle," the target travel speed of the hydraulic excavator 100 can be set faster than when the travel mode is set to "low." When the travel mode is set to "high," for example, the minimum target travel speed is 0 [km / h] and the maximum is 7 [km / h]. When the travel mode is set to "middle," for example, the minimum target travel speed is 0 [km / h] and the maximum is 5 [km / h]. When the driving mode is set to "Low," for example, the minimum target driving speed is 0 [km / h] and the maximum is 3 [km / h].

[0106] The determination unit 25 may lower the relative speed threshold when the target travel speed of the hydraulic excavator 100 is greater than 0 [km / h], in other words, when the hydraulic excavator 100 is traveling, compared to when the target travel speed is 0 [km / h]. In this case, the relative speed threshold VB increases linearly as the target travel speed increases, for example, as shown in Figure 9. In the example shown in Figure 9, the minimum value of the relative speed threshold VB is 1.5 [m / s] and the maximum value is 3 [m / s]. The relative speed threshold VB may increase gradually as the target travel speed increases, for example. The relative speed threshold VB may also increase exponentially as the target travel speed increases, for example.

[0107] The determination unit 25 may set a higher relative speed threshold when the target travel speed of the hydraulic excavator 100 is 0 [km / h], in other words, when the hydraulic excavator 100 is stopped, compared to when the target travel speed is greater than 0 [km / h]. For example, the relative speed threshold VA when the target travel speed of the hydraulic excavator 100 is 0 [km / h] may be higher than 3 [m / s]. The relative speed threshold VA may be a fixed value or may be changeable by the operator.

[0108] The determination unit 25 may lower the relative speed threshold compared to the case where the target travel speed is 0 km / h when the target travel speed of the hydraulic excavator 100 is 0 km / h and the enable / disable switch 124 is operated so that the operation of the hydraulic excavator 100 is enabled.

[0109] The process for setting the relative speed threshold will be explained using Figure 10. When the system 10 for monitoring the area around the hydraulic excavator 100 is started, the process shown in the flowchart in Figure 10 is initiated along with the process shown in the flowchart in Figure 7.

[0110] The first controller 20 determines whether the hydraulic excavator 100 is in a stopped state using the determination unit 25 (step ST31). The determination unit 25 may also determine whether the hydraulic excavator 100 is in a stopped state based on the control command signal output from the work machine operation unit 123. For example, if the first controller 20 has not received a control command signal from the work machine operation unit 123 for a predetermined time, the determination unit 25 may determine that the hydraulic excavator 100 is in a stopped state. For example, if the first controller 20 has received a control command signal from the work machine operation unit 123, the determination unit 25 may determine that the hydraulic excavator 100 is not in a stopped state. The determination unit 25 may also determine whether the hydraulic excavator 100 is in a stopped state based on the control command signal output from the enable / disable switch 124. For example, if the first controller 20 receives a control command signal from the enable / disable switch 124 to disable the operation of the hydraulic excavator 100, the determination unit 25 determines that the hydraulic excavator 100 is in a stopped state. When the first controller 20 receives a control command signal from the enable / disable switching unit 124 to enable the operation of the hydraulic excavator 100, the determination unit 25 determines that the hydraulic excavator 100 is not in a stopped state.

[0111] If the determination unit 25 determines that the hydraulic excavator 100 is stopped (step ST31; Yes), the first controller 20 proceeds to step ST32. If the determination unit 25 determines that the hydraulic excavator 100 is not stopped (step ST31; No), the first controller 20 proceeds to step ST33.

[0112] If the hydraulic excavator 100 is determined to be in a stopped state (step ST31; Yes), the first controller 20 sets the relative speed threshold to the relative speed threshold VA using the determination unit 25 (step ST32). Using this relative speed threshold VA, the process of step ST17 in the flowchart shown in Figure 7 is executed.

[0113] If the hydraulic excavator 100 is determined not to be in a stopped state (step ST31; No), the first controller 20 sets the relative speed threshold to a relative speed threshold VB that changes according to the target travel speed using the determination unit 25 (step ST33). As described above, the relative speed threshold VB is set according to the target travel speed, for example as shown in Figure 9. Using this relative speed threshold VB, the process of step ST17 in the flowchart shown in Figure 7 is executed.

[0114] <Effects> As described above, in this embodiment, when the hydraulic excavator 100 is stopped, the relative speed threshold can be set higher than when it is moving. According to this embodiment, false detection of people or objects can be suppressed. According to this embodiment, people and other objects present around the hydraulic excavator 100 can be detected more accurately.

[0115] In this embodiment, when the hydraulic excavator 100 is moving, a relative speed threshold can be set according to the target travel speed. According to this embodiment, it is possible to reduce the chances of missing detection of people or objects.

[0116] [Third Embodiment] Figure 11 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the third embodiment. In this embodiment, the setting of the threshold value for indicating human-likeness differs from that of the first embodiment. In this embodiment, the threshold value for indicating human-likeness is set instead of a relative speed threshold, which differs from that of the second embodiment. Other aspects are configured the same as in the first and second embodiments.

[0117] The first recognition unit 24 sets a threshold value indicating human-likeness according to the travel speed of the hydraulic excavator 100. In this embodiment, the threshold value indicating human-likeness is described as being set according to the target travel speed of the hydraulic excavator 100. In this embodiment, the threshold value indicating human-likeness includes a first threshold, a second threshold, and a third threshold. At least one of the first threshold, second threshold, and third threshold is set according to the target travel speed of the hydraulic excavator 100. In the following description, it is described as if all of the first threshold, second threshold, and third threshold are set according to the target travel speed.

[0118] The first recognition unit 24 may set a higher threshold for the value indicating human-likeness when the target travel speed of the hydraulic excavator 100 is 0 [km / h], in other words, when the hydraulic excavator 100 is stopped, compared to when the target travel speed is greater than 0 [km / h]. In this case, the threshold values ​​indicating human-likeness may be fixed values, for example, with the first threshold being threshold C1, the second threshold being threshold B1, and the third threshold being threshold A1.

[0119] The first recognition unit 24 may lower the threshold value indicating human-likeness when the target travel speed of the hydraulic excavator 100 is greater than 0 [km / h], in other words, when the hydraulic excavator 100 is traveling, compared to when the target travel speed is 0 [km / h]. In this case, the threshold values ​​indicating human-likeness are set as follows: the first threshold is threshold C2 (C2 < C1), the second threshold is threshold B2 (B2 < B1), and the third threshold is threshold A2 (A2 < A1), and for example, they increase linearly as the target travel speed increases.

[0120] The first recognition unit 24 may change the threshold values ​​indicating human-likeness according to the target travel speed of the hydraulic excavator 100 when the target travel speed is greater than 0 [km / h], in other words, when the hydraulic excavator 100 is traveling. In this case, the threshold values ​​A1, B1, and C1, which are the values ​​indicating human-likeness, may increase in stages as the target travel speed increases, for example. The threshold values ​​A1, B1, and C1 may also increase exponentially as the target travel speed increases, for example.

[0121] The first recognition unit 24 may set a higher threshold value for the human-likeness indicator when the target speed of the hydraulic excavator 100 is 0 [km / h] and the enable / disable switch 124 is operated so that the operation of the hydraulic excavator 100 is disabled, compared to when the target travel speed is greater than 0 [km / h].

[0122] The first recognition unit 24 may lower the threshold value indicating human-likeness compared to the case where the target travel speed of the hydraulic excavator 100 is 0 [km / h] and the enable / disable switch 124 is operated so that the operation of the hydraulic excavator 100 is enabled.

[0123] Using Figure 11, the process of setting a threshold value indicating human-likeness will be explained. When the system 10 for monitoring the area around the hydraulic excavator 100 is started, the process shown in the flowchart in Figure 11 is started along with the process shown in the flowchart in Figure 7. Step ST41 in Figure 11 performs the same process as step ST31 in the flowchart shown in Figure 10. Steps ST42 and ST43 differ from steps ST32 and ST33 in the flowchart shown in Figure 10 in that the processing is performed by the first recognition unit 24.

[0124] If the hydraulic excavator 100 is determined to be in a stopped state (step ST41; Yes), the first controller 20 uses the first recognition unit 24 to set fixed threshold values ​​for the values ​​indicating human-likeness, setting the first threshold to threshold C1, the second threshold to threshold B1, and the third threshold to threshold A1 (step ST42). Using these thresholds A1, B1, and C1 of the values ​​indicating human-likeness, the process of step ST17 in the flowchart shown in Figure 7 is executed.

[0125] If the hydraulic excavator 100 is determined not to be in a stopped state (step ST41; No), the first controller 20 uses the first recognition unit 24 to change the threshold values ​​indicating human-likeness according to the target travel speed, setting the first threshold to threshold C2, the second threshold to threshold B2, and the third threshold to threshold A2 (step ST43). Thresholds A1, B1, and C1 are increased linearly as the target travel speed increases, for example. Using these threshold values ​​indicating human-likeness, thresholds A1, B1, and C1, the processes in steps ST12, ST13, and ST16 of the flowchart shown in Figure 7 are executed.

[0126] <Effects> As described above, in this embodiment, when the hydraulic excavator 100 is stopped, the threshold value indicating human-likeness can be set higher compared to when it is moving. According to this embodiment, false detection of people can be suppressed. According to this embodiment, people and other objects present around the hydraulic excavator 100 can be detected more accurately.

[0127] In this embodiment, when the hydraulic excavator 100 is moving, a threshold value indicating human-likeness can be set according to the target travel speed. According to this embodiment, it is possible to reduce the chances of missing a person from detection.

[0128] [Fourth Embodiment] Figure 12 is a block diagram showing a system for monitoring the surroundings of a work machine according to the fourth embodiment. Figure 13 is a diagram showing the direction of travel of the work machine and the monitoring area. Figure 14 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to the fourth embodiment. In this embodiment, the setting of the relative speed threshold differs from that of the first embodiment. Other aspects are configured the same as in the first embodiment.

[0129] In the hydraulic excavator 100, the slewing body 102 rotates relative to the traveling body 103. Therefore, the direction of travel (direction of movement) of the hydraulic excavator 100 in response to the operation of the travel control lever 123C changes according to the slewing angle. In this embodiment, false detections are more effectively suppressed by changing the relative speed threshold for the monitoring area located in front of the direction of travel of the hydraulic excavator 100.

[0130] The hydraulic excavator 100 is equipped with a travel control lever 123C and a slewing angle sensor 131. The travel control lever 123C and the slewing angle sensor 131 are connected to the first controller 20 in a communication manner.

[0131] The driving operation lever 123C is configured in the same manner as in the second embodiment.

[0132] The rotation angle sensor 131 detects the rotation angle of the traveling body 103 relative to the rotating body 102. The rotation angle sensor 131 is, for example, a potentiometer, a rotation speed pickup sensor, or an inertial measurement unit (IMU). The rotation angle sensor 131 transmits the detected data as an electrical signal to the first controller 20.

[0133] The direction of travel of the traveling body 103 relative to the rotating body 102 is determined by the operating direction of the travel operation lever 123C and the turning angle detected by the turning angle sensor 131.

[0134] Figure 13(a) illustrates the case where the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 0 [°]. When the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 0 [°], the hydraulic excavator 100 moves in the direction of arrow D1 when the travel operation lever 123C is moved forward. When the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 0 [°], the hydraulic excavator 100 moves in the direction of arrow D2 when the travel operation lever 123C is moved backward.

[0135] Figure 13(b) illustrates the case where the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 135°, or in other words, -225°. When the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 135°, the hydraulic excavator 100 moves in the direction of arrow D1 when the travel operation lever 123C is moved forward. When the rotation angle θ of the traveling body 103 relative to the rotating body 102 is 135°, the hydraulic excavator 100 moves in the direction of arrow D2 when the travel operation lever 123C is moved backward.

[0136] The determination unit 25 sets a relative speed threshold used for determining whether a person or object is present, according to the direction of travel of the traveling body 103 of the hydraulic excavator 100. In this embodiment, the determination unit 25 sets the relative speed threshold used for determining whether a person or object is present, according to the operating direction of the travel operation lever 123C and the rotation angle of the traveling body 103 relative to the slewing body 102. More specifically, the determination unit 25 sets the relative speed threshold used for determining whether a person or object is present for each preset monitoring area, based on the operating direction of the travel operation lever 123C and the rotation angle of the traveling body 103 relative to the slewing body 102.

[0137] In this embodiment, the monitoring area consists of a first area and a second area.

[0138] The first region is a region in which the relative velocity threshold is set higher compared to other regions, including the second region. The relative velocity threshold in the first region may be set in the same way as in step ST33 of the flowchart shown in Figure 10.

[0139] The second region is a region in which the relative velocity threshold is set higher compared to other regions, including the first region. The relative velocity threshold in the second region may be set in the same way as in step ST32 of the flowchart shown in Figure 10.

[0140] As shown in Figures 13(a) and 13(b), in this embodiment, the monitoring area is configured to include area 301 and area 302. Areas 301 and 302 include, for example, the range from the end of the vehicle 103 to about a few meters outside in the longitudinal direction of the vehicle 103, perpendicular to the vehicle width direction of the vehicle 103. In the vehicle width direction of the vehicle, the range includes the range from about a few meters radially outside with respect to the turning radius of the vehicle 103. In this embodiment, areas 301 and 302 are set relative to the turning body side coordinate system and are converted to positions in the vehicle side coordinate system based on the turning angle of the vehicle 103 relative to the turning body 102. Areas 301 and 302 may also be set relative to the vehicle side coordinate system.

[0141] The determination unit 25 determines the direction of travel of the traveling body 103 of the hydraulic excavator 100 based on the operating direction of the travel control lever 123C. The determination unit 25 then determines the area set on the side of the hydraulic excavator 100's direction of travel as the first area, based on the direction of travel of the traveling body 103 and the rotation angle of the traveling body 103 relative to the slewing body 102. Furthermore, the determination unit 25 may set the area set on the opposite side of the hydraulic excavator 100's direction of travel as the second area, based on the direction of travel of the traveling body 103 and the rotation angle of the traveling body 103 relative to the slewing body 102.

[0142] The method for determining the region when the slewing angle θ of the hydraulic excavator 100 is 0 [°] will be explained. When the travel control lever 123C is tilted forward, the determination unit 25 determines the region 301 located on the front side of the slewing body 102 as the first region. When the travel control lever 123C is tilted forward, the determination unit 25 may further determine the region 302 located on the rear side of the slewing body 102 as the second region. When the travel control lever 123C is tilted backward, the determination unit 25 determines the region 302 located on the rear side of the slewing body 102 as the first region. When the travel control lever 123C is tilted backward, the determination unit 25 may further determine the region 301 located on the front side of the slewing body 102 as the second region.

[0143] The determination of the region when the slewing angle θ of the hydraulic excavator 100 is 135° will be explained. When the travel control lever 123C is tilted forward, the determination unit 25 determines the region 301 located to the left rear of the slewing body 102 as the first region. When the travel control lever 123C is tilted forward, the determination unit 25 may further determine the region 302 located to the right front of the slewing body 102 as the second region. When the travel control lever 123C is tilted backward, the determination unit 25 determines the region 302 located to the right front of the slewing body 102 as the first region. When the travel control lever 123C is tilted backward, the determination unit 25 may further determine the region 301 located to the left rear of the slewing body 102 as the second region.

[0144] Using Figure 14, the process of setting a relative speed threshold according to the slewing angle will be explained. When the system 10 for monitoring the area around the hydraulic excavator 100 is started, the process shown in the flowchart in Figure 14 is started along with the process shown in the flowchart in Figure 7.

[0145] The first controller 20 determines the direction of travel according to the operating direction of the travel lever 123C and the slewing angle of the hydraulic excavator 100 using the determination unit 25 (step ST51).

[0146] The first controller 20, using the determination unit 25, sets the side of the hydraulic excavator 100 in the direction of travel as the first region and the side opposite to the direction of travel as the second region (step ST52).

[0147] The first controller 20 sets the relative velocity threshold for the first region and the relative velocity threshold for the second region, respectively, using the determination unit 25 (step ST53).

[0148] In step ST53, for example, the relative velocity threshold for the first region is set in the same way as in step ST33 of the flowchart shown in Figure 10. The relative velocity threshold for the second region is set in the same way as in step ST32 of the flowchart shown in Figure 10.

[0149] <Effects> As described above, in this embodiment, the relative speed threshold can be set higher on the side of the hydraulic excavator 100 in the direction of travel compared to when it is moving. According to this embodiment, false detection of people or objects can be suppressed on the side of the hydraulic excavator 100 in the direction of travel. According to this embodiment, people and other objects present around the hydraulic excavator 100 can be detected more accurately.

[0150] In this embodiment, a relative speed threshold can be set on the side of the hydraulic excavator 100 opposite to the direction of travel, according to the target travel speed. According to this embodiment, the detection of people or objects can be suppressed on the side of the hydraulic excavator 100 opposite to the direction of travel.

[0151] [Fifth Embodiment] The setting of the threshold value indicating human-likeness differs from the third embodiment. In this embodiment, the threshold value indicating human-likeness is set instead of the relative speed threshold, which is different from the fourth embodiment. Other aspects are configured the same as in the third and fourth embodiments.

[0152] The first recognition unit 24 sets a threshold value indicating human-likeness according to the slewing angle of the hydraulic excavator 100. In this embodiment, the determination unit 25 sets a threshold value indicating human-likeness according to the operating direction of the travel control lever 123C and the slewing angle of the hydraulic excavator 100. More specifically, the determination unit 25 sets a threshold value indicating human-likeness for each region set according to the operating direction of the travel control lever 123C and the slewing angle of the hydraulic excavator 100.

[0153] The first recognition unit 24 determines the direction of travel of the hydraulic excavator 100's travel body 103 according to the operating direction of the travel control lever 123C and the slewing angle of the hydraulic excavator 100. The first recognition unit 24 then sets the side of the hydraulic excavator 100 in the direction of travel as the first region. Furthermore, the first recognition unit 24 may set the side opposite to the direction of travel of the hydraulic excavator 100 as the second region.

[0154] In this embodiment, the first region may have a threshold value indicating human-likeness set, similar to step ST43 of the flowchart shown in Figure 11. The second region may have a threshold value indicating human-likeness set, similar to step ST42 of the flowchart shown in Figure 11.

[0155] <Effects> As described above, in this embodiment, the threshold value indicating human-likeness can be set higher on the side of the hydraulic excavator 100 in the direction of travel compared to when it is moving. According to this embodiment, false detection of people can be suppressed. According to this embodiment, people and other objects present around the hydraulic excavator 100 can be detected more accurately.

[0156] In this embodiment, a threshold value indicating human-likeness can be set on the side of the hydraulic excavator 100 opposite to the direction of travel, according to the target travel speed. According to this embodiment, it is possible to reduce the chances of missing a person from detection.

[0157] [Modification] In the above-described embodiment, the monitor 15 was described as being located in the operator's cab of the hydraulic excavator 100, but it is not limited to this. If the hydraulic excavator 100 is remotely operated, the monitor 15 may be located in the remote control room.

[0158] In the embodiments described above, the first detection device was described as a camera, but it is not limited to this. The first detection device may be, for example, a LiDAR (Laser Imaging Detection and Ranging).

[0159] In the embodiments described above, the second detection device was described as a radar, but it is not limited to this. The second detection device may be, for example, a stereo camera capable of measuring the distance to an obstacle. The second detection device may also be, for example, a camera that has a short processing time required to detect a person or object.

[0160] In the above-described embodiment, the obstacle processing unit 26, the second recognition unit 27, and the speed acquisition unit 28 were described as being provided in the first controller 20, but these may also be included in the radar 13. In this case, the radar 13 includes a computer system 1000 that implements functions corresponding to the obstacle processing unit 26, the second recognition unit 27, and the speed acquisition unit 28.

[0161] In the above-described embodiment, the relative speed threshold was explained as being set according to the target travel speed of the hydraulic excavator 100, but it is not limited to this. If the hydraulic excavator 100 is equipped with a speed sensor for detecting the travel speed of the hydraulic excavator 100, the relative speed threshold may be set according to the detected travel speed.

[0162] 10...System, 11...Camera (First Detection Device), 13...Radar (Second Detection Device), 15...Monitor (Display Device), 17...Speaker, 20...First Controller, 21...Image Processing Unit, 22...Overhead Image Generation Unit, 23...Image Synthesis Unit, 24...First Recognition Unit, 25...Determination Unit, 26...Obstacle Processing Unit, 27...Second Recognition Unit, 28...Speed ​​Acquisition Unit, 29...Display Control Unit, 40...Second Controller, 41...Work Machine Control Unit, 100...Hydraulic Excavator (Work Machine), 101...Work Machine, 102...Slewing Body, 103...Traction Body, 103C...Track, 106...Boom, 107...Arm, 108...Bucket 109...Blade tip, 110...Hydraulic cylinder, 111...Boom cylinder, 112...Arm cylinder, 113...Bucket cylinder, 120...Operating device, 121...Mode setting unit, 122...Output adjustment unit, 123...Work machine operation unit, 124...Enable / disable switching unit, 200...Overhead image, 1000...Computer system, 1001...Processor, 1002...Main memory, 1003...Storage, 1004...Interface, AX1...Boom axis, AX2...Arm axis, AX3...Bucket axis, AX4...Tilt axis, AX5...Rotate axis, RX...Slewing axis, P...Score.

Claims

1. A system for monitoring the surroundings of a work machine, comprising: a first detection device for detecting the surrounding conditions of the work machine; a second detection device for detecting the surrounding conditions of the work machine in a manner different from that of the first detection device; and a processor, wherein the processor acquires the speed of an obstacle relative to the work machine based on the detection data of the second detection device; determines whether the obstacle included in the detection data of the first detection device is a person based on the detection data of the first detection device and the acquired speed of the obstacle relative to the work machine; and if it is determined that the obstacle is a person, the system notifies that a person has been detected.

2. The system according to claim 1, wherein the processor determines that the obstacle is approaching the work machine at a predetermined speed or higher, and determines that the obstacle included in the detection data of the first detection device is a person.

3. The system according to claim 2, wherein the processor calculates a value indicating human characteristics based on the detection data of the first detection device, and if the value indicating human characteristics is greater than or equal to a first threshold and less than a second threshold greater than the first threshold, and if it is determined that the obstacle is approaching the work machine at a predetermined speed or greater, the processor determines that the obstacle included in the detection data of the first detection device is a person.

4. The system according to claim 3, wherein the processor calculates a value indicating human characteristics based on the detection data of the first detection device, and determines that the obstacle included in the detection data of the first detection device is a person if the value indicating human characteristics is greater than or equal to a second threshold and less than a third threshold greater than the second threshold, and the coordinates of the obstacle included in the detection data of the first detection device and the coordinates of the obstacle detected by the second detection device are within a predetermined range.

5. The system according to claim 4, wherein the processor calculates a value indicating human characteristics based on the detection data of the first detection device, and if the value indicating human characteristics is equal to or greater than a third threshold, it determines that the obstacle included in the detection data of the first detection device is a person, regardless of the detection data of the second detection device.

6. The system according to claim 2, wherein the processor changes a speed threshold for determining that something is a person based on the travel speed of the work machine, and determines that the obstacle included in the detection data of the first detection device is a person when it determines that the obstacle is approaching the work machine at a speed of the speed threshold or higher.

7. The system according to claim 4, wherein the processor modifies at least one of the first threshold, the second threshold, and the third threshold based on the travel speed of the work machine.

8. The system according to claim 2, wherein the processor changes a speed threshold for determining a person in a monitoring area located in front of the direction of travel of the work machine, and determines that the obstacle included in the detection data of the first detection device is a person when it determines that the obstacle in the monitoring area is approaching the work machine at a speed equal to or greater than the speed threshold.

9. The system according to claim 1, wherein the processor controls the work machine when it detects an obstacle detected by the first detection device as a person.

10. The system according to claim 6, wherein the processor controls the work machine by changing the travel speed limit, stopping travel, or stopping turning.

11. The system according to claim 1, wherein the first detection device is a camera and the second detection device is a radar.

12. A method for monitoring the surroundings of a work machine, comprising: a first detection device for detecting the surrounding conditions of the work machine; a second detection device for detecting the surrounding conditions of the work machine in a manner different from that of the first detection device; and a controller having a processor, wherein the method involves: obtaining the speed of an obstacle relative to the work machine based on the detection data of the second detection device; determining whether the obstacle included in the detection data of the first detection device is a person based on the detection data of the first detection device and the obtained speed of the obstacle relative to the work machine; and, if it is determined that the obstacle is a person, notifying that a person has been detected.

13. A work machine comprising: a slewing body; a traveling body supporting the slewing body; a first detection device attached to the slewing body for detecting the surrounding conditions of a work machine; a second detection device attached to the slewing body for detecting the surrounding conditions of the work machine in a manner different from that of the first detection device; and a controller having a processor, wherein the controller acquires the speed of an obstacle relative to the work machine based on the detection data of the second detection device; determines whether the obstacle included in the detection data of the first detection device is a person based on the detection data of the first detection device and the acquired speed of the obstacle relative to the work machine; and if it is determined that the obstacle is a person, it notifies that a person has been detected.

Images