Remote control system for work machine and remote control method for work machine

Abstract

A remote control system for a work machine according to the present invention comprises a processor. The processor acquires a captured image showing a work site of a remotely controlled work machine captured by an image capture device, and displays the captured image on an display device provided separately from the work machine together with a teaching image associated with a work machine operation area specified by a management device for managing the work site.

Description

Remote operation system for construction machinery and method for remotely operating construction machinery

[0001] The present disclosure relates to a remote operation system for construction machinery and a method for remotely operating construction machinery.

[0002] In the technical field related to a remote operation system for construction machinery, a remote operation system as disclosed in Patent Document 1 is known. In the remote operation system, the work site of the construction machinery is imaged by an imaging device. The captured image of the work site captured by the imaging device is displayed on a display device arranged in a remote operation room. An operator in the remote operation room remotely operates the construction machinery while checking the captured image of the work site displayed on the display device.

[0003] Japanese Patent Application Laid-Open No. 2023-174148

[0004] The construction machinery may work in an operation area designated at the work site. When it becomes difficult for an operator in the remote operation room to recognize the operation area, the work efficiency of the construction machinery may decrease.

[0005] An object of the present disclosure is to suppress a decrease in work efficiency.

[0006] According to the present disclosure, a remote operation system for construction machinery is provided. The remote operation system for construction machinery includes a processor. The processor acquires a captured image showing an image of the work site of the remotely operated construction machinery captured by an imaging device, and causes a display device arranged outside the construction machinery to display a teaching image related to the operation area of the construction machinery designated by a management device that manages the work site, together with the captured image.

[0007] According to the present disclosure, a decrease in work efficiency is suppressed.

[0008] Figure 1 is a schematic diagram showing a work site according to the first embodiment. Figure 2 is a schematic diagram showing the management system and remote control system according to the first embodiment. Figure 3 is a schematic side view showing a bulldozer according to the first embodiment. Figure 4 is a schematic diagram showing the soil removal area according to the first embodiment. Figure 5 is a hardware configuration diagram showing the remote controller according to the first embodiment. Figure 6 is a block diagram showing the management system and remote control system according to the first embodiment. Figure 7 is a diagram for explaining the driving data according to the first embodiment. Figure 8 is a diagram for explaining the divided area according to the first embodiment. Figure 9 is a diagram for explaining the dump truck operating area according to the first embodiment. Figure 10 is a diagram for explaining the bulldozer operating area according to the first embodiment. Figure 11 is a diagram for explaining the virtual wall according to the first embodiment. Figure 12 is a diagram for explaining the bulldozer operating area and dump truck operating area according to the first embodiment. Figure 13 is a diagram for explaining the bulldozer operating area and dump truck operating area according to the first embodiment. Figure 14 is a diagram showing the display device according to the first embodiment. Figure 15 is a flowchart showing the display method of the captured image and the taught image according to the first embodiment. Figure 16 shows a display device according to the second embodiment. Figure 17 shows a display device according to the third embodiment. Figure 18 shows a display device according to the fourth embodiment. Figure 19 shows a display device according to the fifth embodiment. Figure 20 is a schematic diagram showing a soil removal area according to the sixth embodiment.

[0009] The embodiments relating to this disclosure will be described below with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [First Embodiment] The first embodiment will now be described.

[0011] <Work Site> Figure 1 is a schematic diagram showing a work site 1 according to the first embodiment. An example of a work site 1 is a mine or a quarry. A mine is a place or business establishment where minerals are extracted. A quarry is a place or business establishment where stone materials are extracted. Examples of mines include metal mines where metals are extracted, non-metallic mines where limestone is extracted, and coal mines where coal is extracted.

[0012] Multiple machines operate at work site 1. Examples of machines include a transport machine that carries cargo, a loading machine that loads cargo onto the transport machine, and an excavator that excavates at least a portion of work site 1. The excavator can perform either or both of the following at work site 1: embankment formation work to form an embankment and / or leveling work to level the ground at work site 1.

[0013] In this embodiment, a dump truck 2, an example of a transport machine, a shovel 3, an example of a loading machine, and a bulldozer 4 and a motor grader 5, examples of excavating machines, operate at the work site 1. The dump truck 2 has a dump body. The dump truck 2 can load cargo onto its dump body and perform transport operations. The shovel 3 has an implement. The shovel 3 can perform loading and excavation operations using its implement. The bulldozer 4 and the motor grader 5 each have an implement. The bulldozer 4 and the motor grader 5 each can perform leveling operations using their implements.

[0014] A loading area 6, a soil removal area 7, and a transport route 8 are provided at the work site 1. Each of the loading area 6, soil removal area 7, and transport route 8 is a work area where the work machinery operates.

[0015] Loading area 6 refers to the work area where loading operations are carried out to load cargo onto dump truck 2. An example of cargo is excavated material excavated in loading area 6. Excavator 3 performs excavation and loading operations in loading area 6.

[0016] The soil removal area 7 is the work area where the soil removal operation is carried out, in which the dump truck 2 unloads its cargo. The bulldozer 4 performs embankment formation and leveling work in the soil removal area 7. The bulldozer 4 may also perform embankment formation and leveling work in the loading area 6.

[0017] The transport path 8 refers to the road on which the dump truck 2 travels. The transport path 8 leads to the loading area 6 and the soil removal area 7, respectively. The transport path 8 is provided to connect at least the loading area 6 and the soil removal area 7. The dump truck 2 traveling towards at least one of the loading area 6 and the soil removal area 7 travels along the transport path 8. The dump truck 2 travels along the transport path 8, for example, by going back and forth between the loading area 6 and the soil removal area 7. The motor grader 5 performs leveling work on the transport path 8.

[0018] Each of the dump truck 2, shovel 3, bulldozer 4, and motor grader 5 may be either an unmanned or manned work machine. An unmanned work machine is a work machine that performs work without operator operation. A manned work machine is a work machine that performs work based on operator operation. In this embodiment, the dump truck 2 is an unmanned work machine. The dump truck 2 travels around the work site 1 unmanned. Each of the shovel 3, bulldozer 4, and motor grader 5 is a manned work machine.

[0019] <Management System and Remote Control System> Figure 2 is a schematic diagram showing the management system 10 and remote control system 19 according to the first embodiment. The management system 10 manages the work site 1. The management system 10 manages the dump truck 2, shovel 3, bulldozer 4, and motor grader 5, respectively. In Figure 2, the dump truck 2 and bulldozer 4 are shown as work machines. In the following description, for the sake of simplicity, an example will be given in which the management system 10 manages the dump truck 2 and bulldozer 4.

[0020] The management system 10 comprises a management device 11 and a communication system 12. The management device 11 includes a computer. The management device 11 is located outside the dump truck 2 and the bulldozer 4. The management device 11 is installed in the control facility 13 of the work site 1. The management device 11 manages the work site 1. The management device 11 manages at least the dump truck 2 and the bulldozer 4. The communication system 12 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN).

[0021] The dump truck 2 comprises a body 14, a running gear 15, a dump body 16, a controller 17, and a wireless communication device 12A. The body 14 includes a body frame. The body 14 is supported by the running gear 15. The running gear 15 supports the body 14 and drives. The running gear 15 includes wheels, tires mounted on the wheels, an engine, a brake system, and a steering system. The dump body 16 is the component into which the cargo is loaded. The dump body 16 is supported by the body 14. The dump body 16 performs dumping and lowering operations. Dumping operation refers to the operation of moving the dump body 16 away from the body 14 and tilting it in the dumping direction. Lowering operation refers to the operation of moving the dump body 16 closer to the body 14. When loading work is performed, the dump body 16 performs a lowering operation. When soil removal work is performed, the dump body 16 performs a dumping operation. The controller 17 includes a computer. The wireless communication device 12A is connected to the controller 17.

[0022] The bulldozer 4 has a controller 18 and a wireless communication device 12B. The controller 18 includes a computer. The wireless communication device 12B is connected to the controller 18.

[0023] The communication system 12 includes a wireless communication device 12A connected to the controller 17 of the dump truck 2, a wireless communication device 12B connected to the controller 18 of the bulldozer 4, and a wireless communication device 12C connected to the management device 11. The management device 11 and the controller 17 of the dump truck 2 communicate wirelessly via the communication system 12. The management device 11 and the controller 18 of the bulldozer 4 communicate wirelessly via the communication system 12.

[0024] In this embodiment, the bulldozer 4 is remotely controlled by a remote control system 19. At least a portion of the remote control system 19 is located in a remote control room 20. The remote control room 20 is located outside the bulldozer 4. The remote control room 20 is installed in a remote location away from the work site 1. The remote control system 19 comprises a remote control device 21, a display device 22, an input device 53, and a remote controller 23.

[0025] The remote control device 21 is located in the remote control room 20. The remote control device 21 is operated by an operator in the remote control room 20. When the remote control device 21 is operated, an operation signal is generated to operate the bulldozer 4. The operator can operate the remote control device 21 while seated in the driver's seat 24.

[0026] The display device 22 is located in the remote control room 20. The display device 22 displays captured images of the work site 1. The display device 22 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). The display device 22 may also be a curved display or may include a projector. The operator operates the remote control device 21 while checking the captured images of the work site 1 displayed on the display device 22. The bulldozer 4 is remotely controlled by the remote control device 21.

[0027] The input device 53 is located in the remote control room 20. The input device 53 generates input data when operated by an operator. Examples of input devices 53 include a touch panel, buttons, and a computer keyboard.

[0028] The remote controller 23 is located in the remote control room 20. The remote controller 23 includes a computer. The remote controller 23 and the controller 18 of the bulldozer 4 communicate via a communication system 25. The communication system 25 includes at least one of the following: the Internet, a mobile phone network, a satellite network, and a local area network (LAN).

[0029] The management device 11 and the remote controller 23 communicate via the communication system 25. Note that the communication system 12 and the communication system 25 may be separate communication systems. At least a portion of the communication system 12 and the communication system 25 may be the same communication system.

[0030] <Bulldozer> Figure 3 is a schematic side view showing a bulldozer 4 according to the first embodiment. As shown in Figure 3, the bulldozer 4 comprises a vehicle body 26, a traveling device 27, an excavating work machine 28, a ripper work machine 29, a position sensor 30, an attitude sensor 31, and an imaging device 32.

[0031] The vehicle body 26 supports an engine (not shown). The engine is the power source for the bulldozer 4. The running gear 27 supports the vehicle body 26 and moves. The running gear 27 has a pair of tracks 33. The bulldozer 4 moves as the tracks 33 rotate.

[0032] The excavation machine 28 performs excavation work, embankment work, soil pushing work, embankment formation work, and land leveling work. The excavation machine 28 is connected to the vehicle body 26. At least a part of the excavation machine 28 is positioned in front of the vehicle body 26. The excavation machine 28 has an excavation blade 34, a lift frame 35, a tilt cylinder 36, and a lift cylinder 37.

[0033] The drilling blade 34 is positioned in front of the vehicle body 26. The drilling blade 34 has a cutting edge 34A. The lift frame 35 supports the drilling blade 34. One end of the lift frame 35 is connected to the back of the drilling blade 34 via a pivot mechanism. The other end of the lift frame 35 is connected to the side of the traveling device 27 via a pivot mechanism. The lift frame 35 includes a first lift frame 35 connected to the left side of the back of the drilling blade 34 and a second lift frame 35 connected to the right side of the back of the drilling blade 34.

[0034] The tilt cylinder 36 and the lift cylinder 37 each operate the drilling blade 34. The tilt cylinder 36 is driven to tilt the drilling blade 34. The lift cylinder 37 is driven to move the drilling blade 34 up and down. One end of the tilt cylinder 36 is connected to the back of the drilling blade 34 via a pivot mechanism. The other end of the tilt cylinder 36 is connected to the upper surface of the lift frame 35. As the tilt cylinder 36 extends and retracts, the tilt angle of the drilling blade 34 changes. One end of the lift cylinder 37 is connected to the back of the drilling blade 34 via a pivot mechanism. The middle part of the lift cylinder 37 is connected to the side of the vehicle body 26. As the lift cylinder 37 extends and retracts, the drilling blade 34 moves in the vertical direction.

[0035] The ripper implement 29 performs ripping operations, such as cutting or crushing the workpiece. For example, it can be used for crushing rocks. The ripper implement 29 is connected to the vehicle body 26. At least a portion of the ripper implement 29 is positioned at the rear of the vehicle body 26. The ripper implement 29 includes a shank 38, a ripper arm 39, a tilt cylinder 40, a lift cylinder 41, and a beam 42.

[0036] The shank 38 is located at the rear of the vehicle body 26. The shank 38 has a ripper point 38A. The ripper point 38A is provided at the tip (lower end) of the shank 38, including the cutting edge position of the ripper. The ripper arm 39 supports the shank 38. The ripper arm 39 connects the vehicle body 26 and the shank 38. One end of the ripper arm 39 is connected to the rear of the vehicle body 26 via a pivot mechanism. The other end of the ripper arm 39 is connected to a beam 42. The beam 42 is rotatably connected to the ripper arm 39. The shank 38 is connected to the ripper arm 39 via the beam 42.

[0037] The tilt cylinder 40 and the lift cylinder 41 each operate the shank 38. The tilt cylinder 40 and the lift cylinder 41 are each connected to the vehicle body 26. The tilt cylinder 40 drives the shank 38 to tilt. The lift cylinder 37 drives the shank 38 to move up and down. One end of the tilt cylinder 40 is connected to the beam 42 via a pivot mechanism. The other end of the tilt cylinder 36 is connected to the rear of the vehicle body 26. As the tilt cylinder 40 extends and retracts, the tilt angle of the shank 38 changes. The tilt cylinder 40 moves the ripper point 38A in the front-rear direction. One end of the lift cylinder 41 is connected to the beam 42 via a pivot mechanism. The other end of the lift cylinder 41 is connected to the rear of the vehicle body 26. As the lift cylinder 41 extends and retracts, the shank 38 moves in the up-and-down direction. The lift cylinder 41 moves the shank 38, which includes the ripper point 38A, in the vertical direction.

[0038] The ripper work machine 29 pierces the workpiece with its ripper point 38A. While the traveling device 27 is stopped, the ripper point 38A is brought into contact with the workpiece, and as the traveling device 27 moves with the ripper point 38A embedded in the workpiece, the workpiece is cut or crushed. While the traveling device 27 is moving, the ripper work machine 29 is moved in the up, down, forward, and backward directions, bringing it into contact with the workpiece, and cutting or crushing occurs as the device moves with the workpiece embedded in it.

[0039] The position sensor 30 detects the position of the bulldozer 4. The position sensor 30 is located on the vehicle body 26. The position sensor 30 detects the position of the bulldozer 4 using the Global Navigation Satellite System (GNSS). The Global Navigation Satellite System includes the Global Positioning System (GPS). The Global Navigation Satellite System detects the position in a global coordinate system defined by latitude, longitude, and altitude coordinate data. A global coordinate system refers to a coordinate system fixed to the Earth. The position sensor 30 includes a GNSS receiver located on the vehicle body 26. The position sensor 30 detects the position of the bulldozer 4 in the global coordinate system.

[0040] The attitude sensor 31 detects the attitude of the bulldozer 4. The attitude sensor 31 detects the attitude of the bulldozer 4 at least when the position sensor 30 has detected the position of the bulldozer 4. The attitude sensor 31 is positioned on the vehicle body 26. The attitude of the bulldozer 4 includes the tilt of the vehicle body 26. The attitude of the bulldozer 4 includes the tilt angle of the vehicle body 26 with respect to the horizontal plane. An inertial measurement unit (IMU) is exemplified as the attitude sensor 31. The attitude sensor 31 is capable of detecting the tilt angle of the vehicle body 26 with respect to the horizontal plane. The attitude of the bulldozer 4 includes the bearing of the bulldozer 4. The bearing of the bulldozer 4 includes the bearing angle with respect to a reference bearing. The attitude sensor 31 is capable of detecting the bearing of the bulldozer 4.

[0041] The imaging device 32 captures images of the object to be imaged. The imaging device 32 is positioned on the vehicle body 26. The object to be imaged by the imaging device 32 includes the work site 1. The object to be imaged by the imaging device 32 includes the ground of the work site 1 surrounding the bulldozer 4 and objects surrounding the bulldozer 4. The images of the object to be imaged captured by the imaging device 32 are displayed on the display device 22 in the remote control room 20. An RGB camera is given as an example of the imaging device 32. However, the imaging device 32 may also be an RGB-D camera or a hyperspectral camera.

[0042] <Dumping Area> FIG. 4 is a diagram schematically showing a dumping area 7 according to the first embodiment. The dumping area 7 is an example of a work area where a dump truck 2 and a bulldozer 4 work at the work site 1. The dump truck 2 can travel in the dumping area 7. The ground of the dumping area 7 is substantially flat. In the dumping area 7, the dump truck 2 performs a dumping operation. There is a cliff 43 around the dumping area 7. The cliff 43 is located outside the edge 44 of the dumping area 7. The cliff 43 is a prohibited area where the work of the dump truck 2 and the bulldozer 4 is prohibited at the work site 1. The dump truck 2 and the bulldozer 4 cannot travel on the cliff 43. The dumping area 7 and the cliff 43 are demarcated by the edge 44 of the dumping area 7. In the embodiment, an embankment 45 is formed on the edge 44. The embankment 45 protrudes upward from the ground of the dumping area 7 near the edge 44.

[0043] In the dumping operation, the dump truck 2 approaches the edge 44 of the dumping area 7 while backing up. After the controller 17 of the dump truck 2 approaches the edge 44 of the dumping area 7, the dump body 16 is dumped. When the dump body 16 is dumped while the dump truck 2 is approaching the edge 44 while backing up, the load loaded on the dump body 16 is dumped onto the cliff 43.

[0044] The height of the embankment 45 is set to a height that does not interfere with the dumping operation of the dump truck 2 and is difficult for the dump truck 2 to climb over the embankment 45. The height of the embankment 45 is, for example, lower than the radius of the tire of the dump truck 2. Since the embankment 45 is formed on the edge 44, it is possible to prevent the dump truck 2 performing the dumping operation from going outside the dumping area 7.

[0045] The bulldozer 4 performs an embankment forming operation for forming the embankment 45 in the dumping area 7. The bulldozer 4 performs a leveling operation for leveling the ground of the dumping area 7. For example, unevenness may be formed on the ground of the dumping area 7 due to the travel of the dump truck 2 or the fall of the load from the dump body 16. The bulldozer 4 performs a leveling operation so that the ground of the dumping area 7 becomes flat.

[0046] <Computer> FIG. 5 is a hardware configuration diagram showing the remote controller 23 according to the first embodiment. The remote controller 23 includes a computer. The management device 11 includes a processor 47 such as a CPU (Central Processing Unit), a main memory 48 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 49, an input / output interface 50 including an input / output circuit, and a communication interface 51 including a communication circuit. The functions of the management device 11 are stored in the storage 49 as a computer program 52. The processor 47 reads the computer program 52 from the storage 49, expands it in the main memory 48, and executes processing according to the computer program 52. Note that the computer program 52 may be distributed to the management device 11 via a network.

[0047] Similar to the remote controller 23, each of the management device 11, the controller 17 of the dump truck 2, and the controller 18 of the bulldozer 4 also includes a computer. Each of the management device 11, the controller 17 of the dump truck 2, and the controller 18 of the bulldozer 4 has a processor, a main memory, a storage for storing a computer program, an input / output interface, and a communication interface.

[0048] FIG. 6 is a block diagram showing the management system 10 and the remote operation system 19 according to the first embodiment.

[0049] The dump truck 2 has a traveling device 15, a dump body 16, a position sensor 55, an azimuth sensor 56, and a speed sensor 57.

[0050] The position sensor 55 detects the position of the dump truck 2. The position sensor 55 includes a GNSS receiver disposed on the dump truck 2. The position sensor 55 detects the position of the dump truck 2 in the global coordinate system.

[0051] The orientation sensor 56 detects the orientation of the dump truck 2. The orientation of the dump truck 2 includes the azimuth angle relative to a reference azimuth. The reference azimuth is, for example, north. An inertial measurement unit (IMU) is exemplified as the orientation sensor 56. The orientation sensor 56 may also include a calculator that calculates the orientation from position data detected by two GNSS antennas installed on the dump truck 2. The calculator can calculate the orientation from a vector connecting the two GNSS antennas. The orientation sensor 56 may also detect the tilt angle of the dump truck 2 with respect to the horizontal plane. If the orientation sensor 56 is an inertial sensor (IMU), it can detect the tilt angle of the dump truck 2 with respect to the horizontal plane.

[0052] The speed sensor 57 detects the travel speed of the dump truck 2. The speed sensor 57 detects the travel speed of the dump truck 2 by, for example, detecting the rotational speed of the drive shaft connected to the wheels of the running gear 15.

[0053] The processor 47 of the remote controller 23 has multiple functional units. The functional units of the processor 47 of the remote controller 23 include an operation signal transmission unit 65, an input data transmission unit 66, an image acquisition unit 67, an operating area data acquisition unit 68, and a display control unit 69.

[0054] The operation signal transmission unit 65 transmits an operation signal generated by the operation of the remote control device 21 to the controller 18 of the bulldozer 4. Based on the operation signal transmitted from the operation signal transmission unit 65, the controller 18 controls at least one of the traveling device 27, the excavating work machine 28, and the ripper work machine 29.

[0055] The input data transmission unit 66 transmits the input data generated by the operation of the input device 53 to the management device 11.

[0056] The image acquisition unit 67 acquires an image showing the work site 1 captured by the imaging device 32. The image acquisition unit 67 also acquires detection data from the position sensor 30 and the attitude sensor 31 when the image is captured by the imaging device 32. The position of an object on the image is defined in the camera coordinate system set in the imaging device 32. Based on the detection data from the position sensor 30 and the attitude sensor 31, the image acquisition unit 67 can convert the position in the global coordinate system to the position on the image in the camera coordinate system.

[0057] The operating area data acquisition unit 68 acquires operating area data relating to the operating area of ​​the bulldozer 4 specified by the management device 11. The management device 11 sets the operating area at the work site 1. The operating area is the area in which the bulldozer 4 should operate.

[0058] The display control unit 69 causes the captured image captured by the imaging device 32 to be displayed on the display device 22. The display control unit 69 also causes the teaching image related to the operating area to be displayed on the display device 22 together with the captured image.

[0059] The processor 47 of the management device 11 has multiple functional units. The functional units of the processor 47 of the management device 11 include a notification data generation unit 58, a soil discharge point setting unit 59, a driving data generation unit 60, a divided area setting unit 61, an operating area designation unit 62, and a virtual wall setting unit 63.

[0060] The notification data generation unit 58 generates notification data to be sent to the operator. The notification data includes display data to be displayed on the display device 22. The notification data generation unit 58 generates display data to be displayed on the display device 22 located outside the bulldozer 4. The notification data generated by the notification data generation unit 58 is transmitted to the remote controller 23.

[0061] The soil discharge point setting unit 59 sets a soil discharge point 72 in the soil discharge area 7. The soil discharge point 72 is the position where the dump truck 2 performs soil discharge work to unload its cargo.

[0062] In this embodiment, the work site 1 is surveyed in advance. Based on the survey data, two-dimensional data representing the two-dimensional shape of the terrain of the work site 1 is calculated. The two-dimensional data represents the two-dimensional shape of the terrain on a predetermined plane substantially parallel to the ground of the work site 1. The two-dimensional data may be defined by latitude and longitude coordinate data. The survey includes detecting the two-dimensional shape of the work site using a position sensor including a GNSS receiver. For example, a survey vehicle equipped with a position sensor travels along the edge of the work area, and the two-dimensional data of the work site 1 is calculated based on the detection data from the position sensor. The two-dimensional data of the work site may be calculated by a management device 11, or by a computer separate from the management device 11.

[0063] Furthermore, three-dimensional data showing the three-dimensional shape of the terrain of the work site 1 may be calculated based on the survey data. The survey may include detecting the three-dimensional shape of the work site using a three-dimensional sensor. A three-dimensional sensor is a sensor capable of detecting the three-dimensional shape of an object to be detected. Examples of three-dimensional sensors include a laser sensor (LiDAR: Light Detection and Ranging) that detects the three-dimensional shape of an object by emitting laser light, and a stereo camera. For example, if an aircraft such as a drone equipped with a three-dimensional sensor flies over the work site 1, and the three-dimensional sensor mounted on the drone detects the work site 1, three-dimensional data of the work site 1 may be calculated based on the detection data of the three-dimensional sensor. The three-dimensional data of the work site may be calculated by the management device 11, or by a computer separate from the management device 11.

[0064] The soil removal point setting unit 59 sets soil removal points 72 in the soil removal area 7 of the work site 1 based on the 3D data of the soil removal area 7. The position of the soil removal points 72 is defined in the global coordinate system. The soil removal point setting unit 59 sets multiple soil removal points 72 in the soil removal area 7.

[0065] The driving data generation unit 60 generates driving data indicating the driving conditions of the dump truck 2 in the work area. The driving data generation unit 60 generates driving data for the dump truck 2 in at least the soil discharge area 7. The driving data generation unit 60 generates driving data for the dump truck 2 based on the soil discharge point 72 set by the soil discharge point setting unit 59. The driving conditions for the dump truck 2 include the target position of the dump truck 2, the target direction of the dump truck 2, the target driving speed of the dump truck 2, and the target turning radius of the dump truck 2.

[0066] Figure 7 is a diagram illustrating the driving data according to the first embodiment. Figure 7 shows the driving data set in the soil removal area 7. The driving data defines the driving conditions of the dump truck 2. The driving data of the dump truck 2 includes the driving point 70, the driving path 79, the target position of the dump truck 2, the target direction of the dump truck 2, and the target driving speed of the dump truck 2. The driving data generation unit 60 generates at least the driving path 79 of the dump truck 2.

[0067] Multiple travel points 70 are set in the soil removal area 7. The travel points 70 define the target position of the dump truck 2. The target position of the dump truck 2 is defined in the global coordinate system. For each of the multiple travel points 70, the target direction and target travel speed of the dump truck 2 are set. The multiple travel points 70 are set at intervals. The intervals between the travel points 70 may be uniform or uneven.

[0068] The target position of dump truck 2 refers to the target position of dump truck 2 when it passes through the travel point 70. The target direction of dump truck 2 refers to the target direction of dump truck 2 when it passes through the travel point 70. The target travel speed of dump truck 2 refers to the target travel speed of dump truck 2 when it passes through the travel point 70. The travel path 79 refers to a virtual line indicating the target travel route of dump truck 2. The travel path 79 is defined by a trajectory that passes through multiple travel points 70. The travel data generation unit 60 generates the travel path 79 so as to include the soil discharge point 72.

[0069] In the example shown in Figure 7, of the multiple travel points 70, one travel point 70 is set as an entry point 70S, and one travel point 70 is set as an exit point 70E. Also, of the multiple travel points 70, at least one travel point 70 is set as a switchback point 71, and at least one travel point 70 is set as an earth removal point 72. The switchback point 71 is the position where the dump truck 2 performs a switchback. A switchback is the action in which the forward-moving dump truck 2 makes a sharp change in direction and moves in reverse.

[0070] The soil removal point 72 may be set near the edge 44 of the soil removal area 7. As described with reference to Figure 4, a mound 45 is formed at the edge 44. The soil removal point 72 may be set at the lower end of the mound 45 on the soil removal area 7 side. The soil removal point 72 may be set on the slope of the mound 45 on the soil removal area 7 side. As described with reference to Figure 4, if the soil removal work is carried out so that a natural bedrock 46 is formed on the ground of the soil removal area 7, the soil removal point 72 may be set at any location in the soil removal area 7 away from the edge 44.

[0071] The location of the travel point 70, which includes the switchback point 71 and the soil removal point 72, and the location of the travel path 79, which indicates the travel route of the dump truck 2, are defined in the global coordinate system.

[0072] The driving data generated in the driving data generation unit 60 is transmitted to the dump truck 2. The controller 17 of the dump truck 2 controls the driving device 15 based on the driving data. The controller 17 controls the driving device 15 so that the dump truck 2 travels according to the driving path 79, based on the detection data from the position sensor 55 and the detection data from the orientation sensor 56. Specifically, the controller 17 controls the driving device 15 so that the deviation between the detected position of the dump truck 2 detected by the position sensor 55 and the target position of the dump truck 2 set at the driving point 70 is reduced when passing through the driving point 70. The controller 17 controls the driving device 15 so that the deviation between the detected orientation of the dump truck 2 detected by the orientation sensor 56 and the target orientation of the dump truck 2 set at the driving point 70 is reduced when passing through the driving point 70. The controller 17 controls the driving device 15 so that the dump truck 2 travels at the target driving speed, based on the detection data from the speed sensor 57. In other words, the controller 17 controls the driving device 15 so that the deviation between the detected driving speed of the dump truck 2 detected by the speed sensor 57 when passing the driving point 70 and the target driving speed of the dump truck 2 set at the driving point 70 becomes small.

[0073] In the example shown in Figure 7, the dump truck 2 enters the soil removal area 7 from the transport path 8 while moving forward. After entering the soil removal area 7 from the transport path 8, the dump truck 2 passes the entry point 70S and then enters the switchback point 71 while moving forward. After entering the switchback point 71, the dump truck 2 performs a switchback and then enters the soil removal point 72 while moving backward. The dump truck 2 that has entered the soil removal point 72 performs soil removal work at the soil removal point 72. With the dump truck 2 positioned at the soil removal point 72, the dump body 16 performs a dumping operation, and the load is discharged from the dump truck 2 at the soil removal point 72. Having completed the soil removal work, the dump truck 2 moves forward towards the exit point 70E to leave the soil removal point 72. After passing the exit point 70E, the dump truck 2 leaves the soil removal area 7 and returns to the transport path 8.

[0074] The division area setting unit 61 sets up multiple division areas 73 in the soil removal area 7 where the bulldozer 4 and dump truck 2 work. The positions of the division areas 73 are defined in the global coordinate system.

[0075] Figure 8 is a diagram illustrating a divided area 73 according to an embodiment. As shown in Figure 8, the soil removal point setting unit 59 sets a plurality of soil removal points 72 in the soil removal area 7. In the example shown in Figure 8, the plurality of soil removal points 72 are set along the edge 44 of the soil removal area 7. In the example shown in Figure 8, soil removal work by the dump truck 2 has not yet been performed. In the following description, soil removal points 72 where soil removal work by the dump truck 2 has not yet been performed will be appropriately referred to as un-soil-removed points 721, and soil removal points 72 where soil removal work by the dump truck 2 has already been performed will be appropriately referred to as soil-removed points 722.

[0076] The division area setting unit 61 sets multiple division areas 73 in the soil removal area 7. The division area setting unit 61 sets the division areas 73 so that the multiple division areas 73 do not overlap with each other. The division area setting unit 61 sets the division areas 73 so that soil removal points 72 are located inside the division areas 73. The division area setting unit 61 sets the division areas 73 so that multiple soil removal points 72 that are adjacent to each other are located inside the division areas 73.

[0077] In the example shown in Figure 8, the division area setting unit 61 sets three division areas 73 in the soil removal area 7. The division area 73 includes a first division area 731, a second division area 732, and a third division area 733. The first division area 731, the second division area 732, and the third division area 733 are set so as not to overlap with each other. The second division area 732 is set next to one side of the first division area 731, and the third division area 733 is set next to the other side of the first division area 731. In the example shown in Figure 8, the first division area 731 includes six adjacent soil removal points 72. The second division area 732 includes seven adjacent soil removal points 72. The third division area 733 includes five adjacent soil removal points 72.

[0078] In the example shown in Figure 8, the outline of the divided area 73 is a rectangle. However, the outline of the divided area 73 is not limited to a rectangle. The outline of the divided area 73 may be any polygon, such as a hexagon or an octagon. The outline of the divided area 73 may also be a circle or an ellipse.

[0079] The division area setting unit 61 determines the size of the division area 73 based on the size of the soil removal area 7. As described above, the work site 1 is surveyed in advance. The size of the soil removal area 7 is calculated based on the survey data. The division area 73 is set inside the soil removal area 7. The size of the division area 73 is smaller than the size of the soil removal area 7. The division area setting unit 61 sets the division area 73 so that the division area 73 is smaller than the soil removal area 7. The division area setting unit 61 sets the division area 73 so that the first division area 731, the second division area 732, and the third division area 733 do not extend beyond the soil removal area 7.

[0080] Each of the bulldozer 4 and dump truck 2 operates within the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 so that each of the bulldozer 4 and dump truck 2 can operate smoothly within the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 so that when each of the bulldozer 4 and dump truck 2 operates within the divided area 73, they are prevented from extending outside the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 based on the vehicle data of the bulldozer 4 and dump truck 2. The vehicle data includes at least one of the external dimensions of the bulldozer 4 and dump truck 2, the minimum turning radius of the bulldozer 4 and dump truck 2, and the external dimensions of the work equipment of the bulldozer 4.

[0081] The external dimensions of the bulldozer 4 and dump truck 2 are known data derived from the specification data of the bulldozer 4 and dump truck 2. The external dimensions of the bulldozer 4 and dump truck 2 include the external dimensions on a predetermined plane parallel to the ground of the soil removal area 7. If the ground of the soil removal area 7 is substantially parallel to a horizontal plane, the external dimensions of the bulldozer 4 and dump truck 2 include the external dimensions in the horizontal direction. The external dimensions of the bulldozer 4 and dump truck 2 may also include the external dimensions in the height direction perpendicular to the ground of the soil removal area 7. The division area setting unit 61 sets the division area 73 such that the larger the external dimensions of the bulldozer 4 and dump truck 2, the larger the division area 73 becomes.

[0082] The minimum turning radius of a work machine refers to the radius of the circle traced by the outermost part of the work machine when it travels across the ground of the work area with the steering angle at its maximum. The minimum turning radius of the bulldozer 4 and dump truck 2 is known data derived from the specifications data of the bulldozer 4 and dump truck 2. The division area setting unit 61 sets the division area 73 such that the larger the turning radius of the bulldozer 4 and dump truck 2, the larger the division area 73 becomes.

[0083] The working equipment of the bulldozer 4 includes an excavating working equipment 28 and a ripper working equipment 29. The external dimensions of the working equipment of the bulldozer 4 are known data derived from the specifications data of the bulldozer 4. The external dimensions of the working equipment of the bulldozer 4 include the external dimensions on a predetermined plane parallel to the ground of the soil removal area 7. The external dimensions of the working equipment of the bulldozer 4 include the external dimensions in the width direction of the bulldozer 4. The external dimensions of the working equipment of the bulldozer 4 are, for example, the width of the excavating blade 34. The external dimensions of the working equipment of the bulldozer 4 may also include the external dimensions in the height direction perpendicular to the ground of the soil removal area 7. The division area setting unit 61 sets the division area 73 such that the larger the external dimensions of the working equipment of the bulldozer 4, the larger the division area 73 becomes.

[0084] The operating area designation unit 62 designates one of the multiple divided areas 73 in which the bulldozer 4 will work. The operating area designation unit 62 also designates one of the multiple divided areas 73 in which the dump truck 2 will work. The divided area 73 in which the bulldozer 4 will work and the divided area 73 in which the dump truck 2 will work are different.

[0085] In the following description, the sectioned area 73 designated for bulldozer 4 to work in will be referred to as the bulldozer work area 74 as appropriate. The sectioned area 73 designated for dump truck 2 to work in will be referred to as the dump truck work area 75 as appropriate. In the bulldozer work area 74, bulldozer 4 works, and dump truck 2 does not. In the dump truck work area 75, dump truck 2 works, and bulldozer 4 does not.

[0086] Figure 9 is a diagram illustrating the dump truck operating area 75 according to the first embodiment. The operating area designation unit 62 designates a divided area 73 that includes an un-excavated soil point 721 from among a plurality of divided areas 73 as the dump truck operating area 75. The driving data generation unit 60 generates a driving path 79 so that soil excavation work is performed at the un-excavated soil point 721 in the dump truck operating area 75. The driving data generation unit 60 generates a driving path 79 so that at least a portion of the driving path 79 is located inside the dump truck operating area 75. The dump truck 2 performs soil excavation work at each of the plurality of un-excavated soil points 721 inside the dump truck operating area 75.

[0087] In the example shown in Figure 9, the first divided area 731 is designated as the dump truck operating area 75. The operating area designation unit 62 designates the first divided area 731 as the dump truck operating area 75 while the bulldozer 4 is located outside the first divided area 731.

[0088] After the soil removal work is completed, the controller 17 of the dump truck 2 transmits a soil removal completion signal to the management device 11, indicating that the soil removal work is finished. The soil removal point setting unit 59 can recognize that the unremoved soil points 721 have changed to soil removed soil points 722 by receiving the soil removal completion signal. Figure 9 shows the state in which the soil removal work of the dump truck 2 is progressing in the first divided area 731, and two of the six unremoved soil points 721 in the first divided area 731 have changed to soil removed soil points 722.

[0089] Figure 10 is a diagram illustrating the bulldozer operating area 74 according to the first embodiment. After all of the un-excavated soil points 721 in the first divided area 731 designated as the dump truck operating area 75 have changed to soil-excavated soil points 722, the operating area designation unit 62 designates the first divided area 731 as the bulldozer operating area 74. That is, the operating area designation unit 62 designates the first divided area 731, where the soil-excavation work of the dump truck 2 has been completed, as the bulldozer operating area 74 where the bulldozer 4 will work. The operating area designation unit 62 also designates a divided area 73, separate from the first divided area 731, as the dump truck operating area 75 where the dump truck 2 will work. In the example shown in Figure 10, the operating area designation unit 62 designates the second divided area 732, which includes the un-excavated soil points 721, as the dump truck operating area 75.

[0090] The notification data generation unit 58 transmits notification data to the remote controller 23 indicating that the first divided area 731 has been designated as the bulldozer operating area 74. The display control unit 69 causes the display device 22 to display display data indicating that the first divided area 731 has been designated as the bulldozer operating area 74.

[0091] The work of bulldozer 4 includes leveling work to smooth the ground in the divided area 73 after the soil removal work has been completed. The operator operates the remote control device 21 so that bulldozer 4 enters the first divided area 731, which has been designated as the bulldozer operating area 74, in order to perform leveling work to smooth the ground in the first divided area 731 after the soil removal work by dump truck 2 has been completed.

[0092] The virtual wall setting unit 63 sets a virtual wall 76 in the first divided area 731 when the bulldozer 4 is inside the first divided area 731 (bulldozer operating area 74) and the dump truck 2 is outside the first divided area 731 (bulldozer operating area 74). The virtual wall setting unit 63 sets the virtual wall 76 in the first divided area 731 after the dump truck 2 has left the first divided area 731 and the bulldozer 4 has entered the first divided area 731. The bulldozer 4 performs leveling work in the first divided area 731 inside the virtual wall 76.

[0093] Figure 11 is a diagram illustrating a virtual wall 76 according to the first embodiment. The virtual wall 76 is a virtual wall set at the boundary between the first divided area 731 and the area outside the first divided area 731 in order to restrict the bulldozer 4 from advancing from the first divided area 731 (bulldozer operating area 74) and the dump truck 2 from entering the first divided area 731 (bulldozer operating area 74). As shown in Figure 11, the virtual wall setting unit 63 sets the virtual wall 76 along the outline of the first divided area 731.

[0094] The virtual wall 76 restricts the operating range of the bulldozer 4. The virtual wall 76 is set to surround the bulldozer 4. The virtual wall 76 is set in the global coordinate system. The virtual wall 76 is set to the south, north, east, west, above, and below the bulldozer 4. The virtual wall 76 may be set to at least one of the south, north, east, west, above, and below the bulldozer 4. The virtual wall 76 does not move even when the bulldozer 4 moves within the work site 1. The virtual wall 76 is fixed within the work site 1. The area outside the virtual wall 76 is an avoidance area where the bulldozer 4 should avoid advancing. When the bulldozer 4 approaches the virtual wall 76, the operation of at least one of the traveling device 27, the excavating machine 28, and the ripper machine 29 is restricted, or an alarm is output in the remote control room 20. In other words, if there is a possibility that at least a part of the bulldozer 4 may go outside the virtual wall 76, the operation of the bulldozer 4 will be restricted or an alarm will be issued to prevent the bulldozer 4 from going outside the virtual wall 76.

[0095] There is a possibility that obstacles may exist at the work site 1 that could hinder the operation of the bulldozer 4. Examples of obstacles include dump trucks 2 traveling in the soil removal area 7, structures above the bulldozer 4, cliffs or holes in the work site 1, and buildings. If at least a part of the bulldozer 4 moves outside the virtual wall 76, the work efficiency of the bulldozer 4 may decrease. By setting a virtual wall 76 at the boundary between the bulldozer operating area 74 where the bulldozer 4 works and the avoidance area, the bulldozer 4 is prevented from moving beyond the virtual wall 76 into the avoidance area. This prevents a decrease in the work efficiency of the bulldozer 4. An example of a virtual wall is disclosed in Japanese Patent Application Publication No. 2024-034408.

[0096] Figure 12 is a diagram illustrating the bulldozer operating area 74 and the dump truck operating area 75 according to the first embodiment. As shown in Figure 12, the first divided area 731 is designated as the bulldozer operating area 74, and the second divided area 732 is designated as the dump truck operating area 75. A virtual wall 76 is set in the first divided area 731 (bulldozer operating area 74).

[0097] After a virtual wall 76 is set in the first divided area 731, the driving data generation unit 60 generates a driving path 79 so that soil removal work is performed at the unremoved soil points 721 in the second divided area 732. The driving data generation unit 60 generates the driving path 79 so that at least a portion of the driving path 79 is located inside the second divided area 732 (dump truck operating area 75). The driving data generation unit 60 generates the driving path 79 so that the dump truck 2 does not enter the first divided area 731 (bulldozer operating area 74). The driving data generation unit 60 generates the driving path 79 so that the driving path 79 does not go inside the virtual wall 76. The dump truck 2 performs soil removal work at each of the multiple unremoved soil points 721 inside the second divided area 732 (dump truck operating area 75).

[0098] After the soil removal work is completed, the controller 17 of the dump truck 2 transmits a soil removal completion signal to the management device 11, indicating that the soil removal work is finished. Upon receiving the soil removal completion signal, the soil removal point setting unit 59 can recognize that the unremoved soil point 721 has changed to a soil removed soil point 722.

[0099] After completing the leveling work in the first divided area 731, the operator of bulldozer 4 operates the input device 53 to notify the management device 11 that the leveling work in the first divided area 731 has been completed. When the input device 53 is operated, input data is generated indicating that the leveling work of bulldozer 4 has been completed. The input data transmission unit 66 of the remote controller 23 transmits the input data generated by the input device 53 to the management device 11.

[0100] The virtual wall setting unit 63 acquires input data indicating that the leveling work of the bulldozer 4 in the first divided area 731 has been completed. After acquiring the input data indicating that the leveling work of the bulldozer 4 has been completed, the virtual wall setting unit 63 determines that the soil removal work of the dump truck 2 in the second divided area 732 has been completed and the dump truck 2 has left the second divided area 732, and then releases the setting of the virtual wall 76. The virtual wall setting unit 63 can determine whether or not the soil removal work of the dump truck 2 in the second divided area 732 has been completed based on the soil removal completion signal from the controller 17 of the dump truck 2. The virtual wall setting unit 63 can determine whether or not the dump truck 2 has left the second divided area 732 based on the detection data from the position sensor 55 of the dump truck 2.

[0101] Figure 13 is a diagram illustrating the bulldozer operating area 74 and the dump truck operating area 75 according to the first embodiment. As shown in Figure 13, the virtual wall setting unit 63 releases the setting of the virtual wall 76 in the first divided area 731 after the bulldozer 4 has finished leveling the ground in the first divided area 731, the dump truck 2 has finished removing soil in the second divided area 732, and the dump truck 2 has left the second divided area 732. By releasing the setting of the virtual wall 76, the bulldozer 4 can move outside the first divided area 731.

[0102] After the setting of the virtual wall 76 is released, the operating area designation unit 62 designates the second divided area 732 as the bulldozer operating area 74 where the bulldozer 4 will perform leveling work. The operating area designation unit 62 designates a divided area 73 from which the dump truck 2 will perform soil removal work, which is different from the first divided area 731 and the second divided area 732. In this embodiment, the third divided area 733 is designated as the dump truck operating area 75.

[0103] The notification data generation unit 58 sends notification data to the remote controller 23 indicating that the setting of the virtual wall 76 in the first divided area 731 has been canceled and that the second divided area 732 has been designated as the bulldozer operating area 74. The display control unit 69 causes the display device 22 to display display data indicating that the setting of the virtual wall 76 in the first divided area 731 has been canceled and that the second divided area 732 has been designated as the bulldozer operating area 74.

[0104] Figure 14 shows a display device 22 according to the first embodiment. As shown in Figure 14, the display control unit 69 causes the display device 22 to display the captured image 80 of the work site 1 captured by the imaging device 32. In the example shown in Figure 14, the captured image 80 displayed on the display device 22 includes a part of the body 26 of the bulldozer 4, the soil removal area 7, and the edge 44 of the soil removal area 7.

[0105] The display control unit 69 generates a reference image 82 showing at least one of the dimensions of the bulldozer 4 and the distance from the bulldozer 4. In one embodiment, the display control unit 69 generates a reference image 82 showing the width of the excavation blade 34 and the distance from the bulldozer 4. The display control unit 69 can generate the reference image 82 based, for example, on the specifications data of the bulldozer 4. The specifications data of the bulldozer 4 is known data.

[0106] The reference image 82 is superimposed on the captured image 80. The reference image 82 shows the width of the excavation blade 34 and the distance from the bulldozer 4. The reference image 82 includes a blade reference line 821 showing the width of the excavation blade 34 and a distance reference line 822 showing the distance from the bulldozer 4. The blade reference line 821 is displayed so as to extend forward from the left end and the right end of the excavation blade 34, respectively. The left blade reference line 821 and the right blade reference line 821 are parallel. The distance reference line 822 is displayed so as to connect the front end of the left blade reference line 821 and the front end of the right blade reference line 821. The distance reference line 822 is displayed at a position a specified distance (e.g., 20 m) forward from the front end of the vehicle body 26.

[0107] In this embodiment, the display control unit 69 causes the display device 22 to display a teaching image 90 relating to the bulldozer operating area 74 of the bulldozer 4. The display control unit 69 displays the teaching image 90 together with the captured image 80 on the display device 22. As shown in Figure 14, the display control unit 69 superimposes the teaching image 90 onto the captured image 80.

[0108] As explained with reference to Figure 13, after completing the leveling work in the first divided area 731, the bulldozer 4 moves outside the first divided area 731. The display control unit 69 causes the display device 22 to display the teaching image 90 when the bulldozer 4 moves outside at least the first divided area 731 (bulldozer operating area 74).

[0109] The control device 11 manages multiple bulldozer operating areas 74 of the bulldozer 4. As explained with reference to Figure 13, after the bulldozer 4 has finished leveling the ground in the first divided area 731, the operating area designation unit 62 designates the second divided area 732 as the bulldozer operating area 74 where the bulldozer 4 will perform leveling work. The bulldozer 4 then performs leveling work in the second divided area 732 after the first divided area 731.

[0110] The operating area data acquisition unit 68 acquires operating area data from the management device 11 indicating the bulldozer operating area 74 in which the bulldozer 4 will perform land leveling work next after completing land leveling work in the first divided area 731. The display control unit 69 causes the display device 22 to display a teaching image 90 indicating that the bulldozer operating area 74 in which land leveling work will be performed next after the first divided area 731 is the second divided area 732. The teaching image 90 is a guidance image that teaches the divided area 73 in which the bulldozer 4 will perform land leveling work next.

[0111] In the example shown in Figure 14, the bulldozer 4, having completed the leveling work in the first divided area 731, is located in the first divided area 731. When the bulldozer 4 completes the leveling work in the first divided area 731 (bulldozer operating area 74), the display control unit 69 causes the display device 22 to display a teaching image 90 indicating the second divided area 732, where the leveling work will be performed next.

[0112] The teaching image 90 includes a position image 91 that shows the location of the second divided area 732. The position image 91 is a graphic image that shows a specific location in the second divided area 732. The specific location is, for example, the center of the second divided area 732. In the example shown in Figure 14, the position image 91 is a star-shaped image displayed at the center of the second divided area 732. In the example shown in Figure 14, the display control unit 69 causes the display device 22 to display an excavation point image 72V that shows the excavation point 72 of the second divided area 732. The specific location where the position image 91 is displayed may be the excavation point 72 or a vicinity of the excavation point 72.

[0113] The teaching image 90 includes a compass image 92 indicating the direction from the bulldozer in the first divided area 731 to the second divided area 732. The compass image 92 is a graphic image indicating the direction of travel of the bulldozer 4. In the example shown in Figure 14, the compass image 92 is an arrow image extending from the front of the bulldozer 4's body 26 toward the center of the second divided area 732.

[0114] When the teaching image 90 is displayed on the display device 22, the operator can recognize the bulldozer operating area 74 where work should be performed next. The operator operates the remote control device 21 so that the bulldozer 4 enters the second divided area 732, which has been designated as the bulldozer operating area 74, in order to perform leveling work to flatten the ground in the second divided area 732 where the soil removal work has been completed.

[0115] After the bulldozer 4 enters the second divided area 732, the display control unit 69 terminates the display of the teaching image 90.

[0116] The display control unit 69 displays the teaching image 90 on the display device 22 at a predetermined display timing. When the display control unit 69 guides the bulldozer 4 to move from the first divided area 731 to the second divided area 732, it may display the teaching image 90 on the display device 22 when the leveling work of the bulldozer 4 in the first divided area 731 is completed. As described above, the operator operates the input device 53 after the leveling work in the first divided area 731 is completed. The display control unit 69 may display the teaching image 90 on the display device 22 at the time input data is generated by the operation of the input device 53. The display control unit 69 may also display the teaching image 90 on the display device 22 before the second divided area 732 is designated as the bulldozer operating area 74. The display control unit 69 may display the teaching image 90 on the display device 22 when the second divided area 732 is designated as the bulldozer operating area 74 and the leveling work of the bulldozer 4 in the first divided area 731 is completed.

[0117] The display control unit 69 may display the teaching image 90 on the display device 22 when the leveling work of the bulldozer 4 in the first divided area 731 is completed and the bulldozer 4 transitions to a state where it can move from the first divided area 731 to the second divided area 732. The state in which the bulldozer 4 can move from the first divided area 731 to the second divided area 732 includes the state in which the soil removal work of the dump truck 2 in the second divided area 732 is completed.

[0118] The display control unit 69 may display the teaching image 90 on the display device 22 after the bulldozer 4 has finished leveling the ground in the first divided area 731 and the bulldozer 4 has moved outside the first divided area 731.

[0119] As explained with reference to Figure 13, in this embodiment, the state in which the bulldozer 4 can move from the first divided area 731 to the second divided area 732 includes the state in which the bulldozer 4 has finished leveling the ground in the first divided area 731, the dump truck 2 has finished removing soil in the second divided area 732, the dump truck 2 has left the second divided area 732, and the setting of the virtual wall 76 in the first divided area 731 has been released.

[0120] <Display Method> Figure 15 is a flowchart showing the display method of the captured image 80 and the teaching image 90 according to the first embodiment. The image acquisition unit 67 acquires the captured image 80 of the work site 1 captured by the imaging device 32 (step S1). The display control unit 69 displays the captured image 80 on the display device 22 (step S2).

[0121] The operating area data acquisition unit 68 determines whether the second divided area 732 has been designated as the bulldozer operating area 74 (step S3).

[0122] When the leveling work of the bulldozer 4 in the first divided area 731 is completed, the operating area designation unit 62 of the control device 11 designates the second divided area 732 as the bulldozer operating area 74. The notification data generation unit 58 transmits notification data to the remote controller 23 indicating that the second divided area 732 has been designated as the bulldozer operating area 74. The operating area data acquisition unit 68 can determine whether or not the second divided area 732 has been designated as the bulldozer operating area 74 based on the notification data from the control device 11.

[0123] In step S3, if it is determined that the second divided area 732 has been designated as the bulldozer operating area 74 (step S3: Yes), the operating area data acquisition unit 68 acquires operating area data indicating that the bulldozer operating area 74, which will be leveled after the first divided area 731, is the second divided area 732 (step S4).

[0124] The display control unit 69 determines whether or not it is the display timing to display the teaching image 90 after the operating area data acquisition unit 68 has acquired the operating area data (step S5).

[0125] In step S5, if it is determined that it is time to display the teaching image 90 (step S5: Yes), the display control unit 69 superimposes the teaching image 90 onto the captured image 80 (step S6).

[0126] In step S3, if it is determined that the bulldozer operating area 74 is not specified (step S3: No), the display control unit 69 displays the captured image 80 on the display device 22 and does not display the teaching image 90. In step S5, if it is determined that it is not the timing to display the teaching image 90 (step S5: No), the display control unit 69 displays the captured image 80 on the display device 22 and does not display the teaching image 90.

[0127] The display control unit 69 determines whether or not to terminate the display of the captured image 80 on the display device 22 (step S7). If it is determined in step S7 to continue displaying the captured image 80 (step S7: No), the process returns to step S1. If it is determined in step S7 to terminate the display of the captured image 80 (step S7: Yes), the display of the captured image 80 is terminated.

[0128] <Effects> As described above, the processor 47 of the remote controller 23 includes an image acquisition unit 67 that acquires an image 80 showing an image of the work site 1 of the remotely operated bulldozer 4 captured by the imaging device 32, and a display control unit 69 that causes a teaching image 90 relating to the bulldozer operating area 74, which is the operating area of ​​the bulldozer 4 designated by the management device 11 that manages the work site 1, to be displayed together with the image 80 on a display device 22 located outside the bulldozer 4.

[0129] According to this embodiment, the teaching image 90 relating to the bulldozer operating area 74 is displayed on the display device 22 together with the captured image 80, so that the operator in the remote control room 20 can recognize the bulldozer operating area 74. As a result, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0130] The control device 11 manages multiple divided areas 73 that may be designated as the bulldozer operating area 74. The display control unit 69 displays a teaching image 90 that shows the second divided area 732, which will be designated as the bulldozer operating area 74 after the first divided area 731. By checking the teaching image 90 displayed on the display device 22, the operator can recognize the second divided area 732, which should be leveled after the first divided area 731.

[0131] The teaching image 90 includes a position image 91 that shows the location of the second division area 732, which is designated as the bulldozer operating area 74 after the first division area 731. The operator can recognize the location of the second division area 732 by checking the position image 91 displayed on the display device 22. Since the position image 91 is superimposed on the captured image 80, the operator can recognize the relative position between the bulldozer 4 and the second division area 732 based on the relative positions of the captured image 80 and the position image 91 of the bulldozer 4 displayed on the display device 22.

[0132] The teaching image 90 includes a compass image 92 that shows the direction from the bulldozer 4 located in the first divided area 731 to the second divided area 732, which is designated as the bulldozer operating area 74 after the first divided area 731. By checking the compass image 92 displayed on the display device 22, the operator can recognize the direction in which the bulldozer 4 should move.

[0133] The display control unit 69 starts displaying the teaching image 90 at any display timing and ends displaying the teaching image 90 at any end timing. For example, the display control unit 69 starts displaying the teaching image 90 when the bulldozer 4 finishes work in the first divided area and ends displaying the teaching image 90 after the bulldozer 4 enters the second divided area 732. Since the teaching image 90 is displayed only for the period for which it is necessary, the operator is less likely to feel annoyed.

[0134] The display control unit 69 may also display the teaching image 90 when the bulldozer 4 has finished its work in the first divided area 731 and has transitioned to a state where the bulldozer 4 can move from the first divided area 731 to the second divided area 732. When the bulldozer 4 is in the first divided area 731, the dump truck 2 performs soil removal work in the second divided area 732. The state in which the bulldozer 4 can move from the first divided area 731 to the second divided area 732 may include the state in which the soil removal work of the dump truck 2 in the second divided area 732 has been completed. After the soil removal work of the dump truck 2 is completed and the dump truck 2 has left the second divided area 732, the operator has the bulldozer 4 enter the second divided area 732 based on the teaching image 90 displayed on the display device 22. This prevents contact between the dump truck 2 and the bulldozer 4.

[0135] The work of bulldozer 4 in the first divided area 731 includes leveling the ground in the first divided area 731. The ground in the soil removal area 7 is leveled by bulldozer 4.

[0136] [Second Embodiment] A second embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0137] Figure 16 shows a display device 22 according to the second embodiment. As shown in Figure 16, leveling work is performed by a bulldozer 4 in the first divided area 731. In the example shown in Figure 16, the third divided area 733 is designated as the bulldozer operating area 74 after the first divided area 731. The display control unit 69 causes the display device 22 to display a position image 91 indicating the position of the third divided area 733.

[0138] In this embodiment, the teaching image 90 includes a path image 93 showing the travel path from the bulldozer 4 located in the first divided area 731 to the third divided area 733.

[0139] As described above, three-dimensional data of the work site 1 is calculated in advance based on survey data. The management device 11 manages the three-dimensional data of the work site 1. The operating area data acquisition unit 68 can generate a travel path for the bulldozer 4 from the bulldozer 4 located in the first divided area 731 to the third divided area 733 based on the three-dimensional data of the soil removal area 7. The operating area data acquisition unit 68 can generate the travel path for the bulldozer 4 based on an existing path search algorithm, such as the A* search algorithm. The display control unit 69 generates a path image 93 along the travel path of the bulldozer 4. The path image 93 is a line-shaped image along the travel path of the bulldozer 4. The display control unit 69 overlays the path image 93 on the captured image 80. The path image 93 is displayed on the display device 22 so as to connect the body 26 of the bulldozer 4 and the position image 91 in the captured image 80.

[0140] The control device 11 manages the location of obstacles 77 that obstruct the movement of the bulldozer 4 at the work site 1. The operating area data acquisition unit 68 generates a travel path for the bulldozer 4 from the bulldozer 4 in the first divided area 731 to the third divided area 733, based on the three-dimensional data of the excavation area 7 including the obstacles 77. The display control unit 69 generates a path image 93 that bypasses the obstacles 77. The path image 93 is displayed on the display device 22 so as to bypass the obstacles 77 in the captured image 80.

[0141] As described above, according to this embodiment, the path image 93 is displayed on the display device 22 together with the captured image 80, so that the operator in the remote control room 20 can recognize the travel path of the bulldozer 4. Therefore, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0142] [Third Embodiment] A third embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0143] Figure 17 shows a display device 22 according to the third embodiment. As shown in Figure 17, the display control unit 69 may display an overhead image 7V of the soil removal area 7 on the display device 22. As described above, the work site 1 is surveyed and three-dimensional data of the work site 1 is calculated. The notification data generation unit 58 can generate an overhead image 7V of the soil removal area 7 based on the three-dimensional data of the work site 1. The display control unit 69 displays the overhead image 7V generated by the notification data generation unit 58 on the display device 22.

[0144] The display control unit 69 causes the display device 22 to display a bulldozer image 4V, which represents a model of the bulldozer 4. The bulldozer image 4V is an image that mimics the bulldozer 4 as seen from above. When the bulldozer 4 travels in the excavation area 7, the bulldozer image 4V moves on the display screen of the display device 22 in accordance with the movement of the bulldozer 4. The display control unit 69 can move the bulldozer image 4V on the display screen of the display device 22 based on the detection data from the position sensor 30 and attitude sensor 31 of the bulldozer 4.

[0145] The display control unit 69 causes the display device 22 to display a first line image 731V showing the outline of the first divided area 731, a second line image 732V showing the outline of the second divided area 732, and a third line image 733V showing the outline of the third divided area 733.

[0146] The display control unit 69 causes the display device 22 to display a bulldozer operating area image 74V, which shows the bulldozer operating area 74, and a dump truck operating area image 75V, which shows the dump truck operating area 75. In the example shown in Figure 17, the bulldozer operating area image 74V is a pattern image that is displayed by filling in the inside of the first line image 731V and the inside of the second line image 732V, respectively. The dump truck operating area image 75V is a pattern image that is displayed by filling in the inside of the third line image 733V. The display format of the bulldozer operating area image 74V and the display format of the dump truck operating area image 75V are different. By checking the display device 22, the operator can recognize that the first divided area 731 and the second divided area 732 have been designated as the bulldozer operating area 74, and that the third divided area 733 has been designated as the dump truck operating area 75.

[0147] The display control unit 69 causes the display device 22 to display an un-excavated point image 721V indicating an unexcavated point 721 and an excavated point image 722V indicating an excavated point 722. The display format of the un-excavated point image 721V and the display format of the excavated point image 722V are different. As described above, after the excavation work of the dump truck 2 is completed, the controller 17 of the dump truck 2 transmits an excavation completion signal to the management device 11 indicating that the excavation work has been completed. Upon receiving the excavation completion signal, the notification data generation unit 58 can generate display data to be displayed on the display device 22 so that the un-excavated points 721 and the excavated points 722 are displayed in different formats. By checking the display data displayed on the display device 22, the operator can confirm the location and number of un-excavated points 721 and the location and number of excavated points 722.

[0148] The display control unit 69 causes the display device 22 to display, as a teaching image 90, the second division area 732 which is designated as the bulldozer operating area 74 after the first division area 731, as a teaching image 90, which includes a position image 91 showing the location of the second division area 732 and an orientation image 92 showing the direction from the bulldozer 4 in the first division area 731 to the second division area 732. The position image 91 is displayed at the center of the rectangular second line image 732V. The orientation image 92 is displayed on the display device 22 so as to extend from the first line image 731V showing the first division area 731 to the second line image 732V showing the second division area 732.

[0149] As described above, in this embodiment as well, the teaching image 90 is displayed on the display device 22 together with the overhead image 7V, so that the operator in the remote control room 20 can recognize the bulldozer operating area 74. Therefore, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0150] [Fourth Embodiment] A fourth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0151] Figure 18 shows a display device 22 according to the fourth embodiment. In this embodiment, a plurality of display devices 22 are arranged side by side in the remote control room 20. In the example shown in Figure 18, the display device 22 includes a first display device 22A and a second display device 22B arranged next to the first display device 22A.

[0152] In this embodiment, the display control unit 69 causes the first display device 22A to display the captured image 80 and the second display device 22B to display the teaching image 90. In the example shown in Figure 18, the captured image 80 displayed on the first display device 22A includes a part of the body 26 of the bulldozer 4, the dump truck 2, the soil discharge area 7, and the edge 44 of the soil discharge area 7. The teaching image 90 displayed on the second display device 22B includes a position image 91 and an orientation image 92. The second display device 22B displays an overhead view of the soil discharge area 7. The teaching image 90 is superimposed on the overhead view of the soil discharge area 7.

[0153] [Fifth Embodiment] The fifth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0154] Figure 19 shows a display device 22 according to the fifth embodiment. As shown in Figure 19, the display control unit 69 may generate a window image in which the teaching image 90 is superimposed on an overhead view image of the soil removal area 7, and then superimpose the generated window image onto the captured image 80. Superimposition display refers to a display method in which a window image, which is a secondary screen, is superimposed on a part of the captured image 80, which is the main screen.

[0155] [Sixth Embodiment] The sixth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0156] Figure 20 is a schematic diagram showing the soil removal area 7 according to the sixth embodiment. As shown in Figure 20, the dump truck 2 may perform soil removal work so that multiple piles 46 are formed on the ground of the soil removal area 7. The multiple piles 46 are formed regularly at intervals in the soil removal area 7. The bulldozer 4 can perform leveling work to smooth out the multiple piles 46 formed on the ground of the soil removal area 7. Even when multiple piles 46 are formed on the ground of the soil removal area 7, the display control unit 69 can display the teaching image 90 related to the bulldozer operating area 74 together with the captured image 80 on the display device 22.

[0157] [Other Embodiments] In the above-described embodiment, the reference image 82 is superimposed on the captured image 80. The reference image 82 does not need to be displayed.

[0158] In the above embodiment, a virtual wall 76 is set in the bulldozer operating area 74. A virtual wall 76 does not have to be set.

[0159] In the above embodiment, the reference image 82 does not need to be displayed on the display device 22.

[0160] In the above embodiment, the remotely controlled work machine is a bulldozer 4. The remotely controlled work machine may be other work machines such as an excavator, wheel loader, and motor grader.

[0161] In the above-described embodiment, at least a portion of the functional parts of the remote controller 23 may be provided in the management device 11. At least a portion of the functional parts of the management device 11 may be provided in the remote controller 23. For example, at least one of the notification data generation unit 58, soil discharge point setting unit 59, travel data generation unit 60, division area setting unit 61, operating area designation unit 62, and virtual wall setting unit 63 may be provided in the remote controller 23.

[0162] In the above-described embodiment, each of the functional units of the management device 11 may be configured by a separate computer (hardware). Each of the multiple functional units of the remote controller 23 may be configured by a separate computer (hardware). The multiple functional units of the management device 11 and the multiple functional units of the remote controller 23 may be configured by a single computer (hardware).

[0163] 1...Work site, 2...Dump truck, 3...Excavator, 4...Bulldozer (working machine), 4V...Bulldozer image, 5...Motor grader, 6...Loading area, 7...Soil removal area, 7V...Overhead view image, 8...Transport route, 10...Management system, 11...Management device, 12...Communication system, 12A...Wireless communication device, 12B...Wireless communication device, 12C...Wireless communication device, 13...Control facility, 14...Vehicle body, 15...Running gear, 16...Dump body, 17...Controller, 18...Controller, 19...Remote control system, 20...Remote control room, 21...Remote control device, 22...Display device, 22A...First display device 22B...Second display device, 23...Remote controller, 24...Driver's seat, 25...Communication system, 26...Vehicle body, 27...Running gear, 28...Excavating work machine, 29...Ripper work machine, 30...Position sensor, 31...Attitude sensor, 32...Imaging device, 33...Track, 34...Excavating blade, 34A...Cutting edge, 35...Lift frame, 36...Tilt cylinder, 37...Lift cylinder, 38...Shank, 38A...Ripper point, 39...Ripper arm, 40...Tilt cylinder, 41...Lift cylinder, 42...Beam, 43...Downhill cliff, 44...Edge, 45...Embankment, 46...Ground, 47...Pro 48...Main memory, 49...Storage, 50...Input / output interface, 51...Communication interface, 52...Computer program, 53...Input device, 55...Position sensor, 56...Direction sensor, 57...Speed ​​sensor, 58...Notification data generation unit, 59...Soil removal point setting unit, 60...Travel data generation unit, 61...Division area setting unit, 62...Operation area specification unit, 63...Virtual wall setting unit, 65...Operation signal transmission unit, 66...Input data transmission unit, 67...Image acquisition unit, 68...Operation area data acquisition unit, 69...Display control unit, 70...Travel point, 70E...Departure point, 70S...Move Entry point, 71... Switchback point, 72... Soil removal point, 72V... Soil removal point image, 73... Divided area, 74... Bulldozer operating area, 74V... Bulldozer operating area image, 75... Dump truck operating area, 75V... Dump truck operating area image, 76... Virtual wall, 79... Driving path, 80... Captured image, 82... Reference image, 90... Teaching image, 91... Position image, 92... Orientation image, 93... Path image, 721... Unremoved point, 721V... Unremoved point image, 722... Soil removal completed point, 722V... Soil removal completed point image, 731... First divided area, 731V... First line image, 732... Second divided area,732V...Second line image, 733...Third divided area, 733V...Third line image, 821...Blade guideline, 822...Distance guideline.

Claims

1. A remote control system for a work machine, comprising a processor, wherein the processor acquires an image showing a work site of a remotely controlled work machine captured by an imaging device, and displays a teaching image relating to the operating area of ​​the work machine, specified by a management device that manages the work site, together with the image on a display device located outside the work machine.

2. The remote control system for a work machine according to claim 1, wherein the management device manages a plurality of work areas, and the processor displays the teaching image indicating the second work area that the work machine will then work on after completing work in the first work area.

3. The remote control system for a work machine according to claim 2, wherein the teaching image includes a position image indicating the location of the second working area.

4. The remote control system for a work machine according to claim 2, wherein the teaching image includes an azimuth image indicating the direction from the work machine located in the first working area to the second working area.

5. The remote control system for a work machine according to claim 2, wherein the teaching image includes a path image showing a travel path from the work machine located in the first working area to the second working area.

6. The remote control system for a work machine according to claim 5, wherein the management device manages the location of obstacles present at the work site that obstruct the movement of the work machine, and the path image is generated to bypass the obstacles.

7. The remote control system for a work machine according to claim 2, wherein the processor displays the teaching image when the work of the work machine in the first operating area is completed.

8. The remote control system for a work machine according to claim 2, wherein the processor displays the teaching image when the work of the work machine in the first working area is completed and the work machine transitions to a state where it can move to the second working area.

9. The remote control system for a work machine according to claim 8, wherein the state in which an unmanned vehicle performs soil removal work in the second operating area and the work machine is movable to the second operating area includes the state in which the soil removal work of the unmanned vehicle in the second operating area is completed.

10. The remote control system for a work machine according to claim 9, wherein the work of the work machine in the first working area includes either or both of the following: embankment formation work for forming an embankment in the first working area and ground leveling work for leveling the ground in the first working area.

11. The remote control system for a work machine according to claim 2, wherein the processor displays the teaching image after the work of the work machine in the first operating area has been completed and the work machine has moved outside the first operating area.

12. The remote control system for a work machine according to claim 1, wherein the management device sets up a plurality of divided areas at the work site and designates the working area from the plurality of divided areas.

13. The remote control system for a work machine according to claim 1, wherein the processor overlays the teaching image onto the captured image.

14. A remote control system for a work machine according to claim 1, wherein a remote control device for remotely operating the work machine is located in a remote control room, a plurality of the display devices are arranged side by side, and the processor causes the first display device to display the captured image and the second display device to display the teaching image.

15. A method for remotely operating a work machine, comprising: acquiring an image showing an image of the work site of a remotely operated work machine captured by an imaging device; and displaying a teaching image relating to the operating area of ​​the work machine, specified by a management device that manages the work site, together with the image on a display device located outside the work machine.

16. The remote control method for a work machine according to claim 15, wherein the management device manages a plurality of work areas and displays the teaching image indicating the second work area that the work machine will then work on after completing work in the first work area.

17. The remote control method for a work machine according to claim 16, wherein the teaching image includes a position image indicating the location of the second working area.

18. The remote control method for a work machine according to claim 16, wherein the teaching image includes an azimuth image indicating the direction from the work machine located in the first working area to the second working area.

19. The remote control method for a work machine according to claim 16, wherein the teaching image includes a path image showing a travel path from the work machine located in the first working area to the second working area.

20. The remote control method for a work machine according to claim 19, wherein the management device manages the location of obstacles present at the work site that obstruct the movement of the work machine, and the path image is generated to bypass the obstacles.

Images