Remote operation system for work machine and remote operation method for work machine

Abstract

This remote operation system for a work site comprises a processor. The processor acquires three-dimensional data indicating a three-dimensional shape detected for the work site by a three-dimensional sensor of a first work machine working at the work site, generates a three-dimensional image of the work site on the basis of the three-dimensional data, and generates display data that includes the three-dimensional image and is to be displayed by a display device disposed outside the first work machine.

Description

Remote control system for working machine and remote control method for working machine

[0001] The present disclosure relates to a remote control system for a working machine and a remote control method for a working machine.

[0002] In the technical field related to remote control systems for working machines, a remote control system as disclosed in Patent Document 1 is known.

[0003] Japanese Unexamined Patent Publication No. 2018-207244

[0004] In a remote control system, images of a work site where a working machine operates are displayed on a display device disposed in a remote control room. An operator operates the remote control device while checking the images of the work site displayed on the display device. If the operator cannot properly grasp the situation of the work site, work efficiency may decrease.

[0005] An object of the present disclosure is to suppress a decrease in work efficiency.

[0006] According to the present disclosure, there is provided a remote control system for a work site including a processor. The processor acquires three-dimensional data indicating a three-dimensional shape of the work site detected by a three-dimensional sensor included in a first working machine operating at the work site, generates a three-dimensional image of the work site based on the three-dimensional data, and generates display data including the three-dimensional image to be displayed on a display device disposed outside the first working machine.

[0007] According to the present disclosure, a decrease in work efficiency is suppressed.

[0008] Figure 1 is a schematic diagram showing a work site management system according to the embodiment. Figure 2 is a schematic side view showing a work machine according to the embodiment. Figure 3 is a schematic top view showing a work machine according to the embodiment. Figure 4 is a hardware configuration diagram showing a remote controller according to the embodiment. Figure 5 is a functional block diagram showing a management system according to the embodiment. Figure 6 is a diagram for explaining the operation of an aircraft according to the embodiment. Figure 7 is a diagram showing an example of display data shown on a display device according to the embodiment. Figure 8 is a diagram showing an example of display data shown on a display device according to the embodiment. Figure 9 is a flowchart showing a method for displaying display data according to the embodiment. Figure 10 is a diagram showing an example of a display device according to the embodiment.

[0009] The embodiments of this disclosure will be described below with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [Management System] Figure 1 is a schematic diagram showing the management system 2 of a work site 1 according to an embodiment. A mine or quarry is an example of a work site 1. A mine is a place or business establishment where minerals are extracted. A quarry is a place or business establishment where stone materials are extracted. Examples of mines include metal mines where metals are extracted, non-metallic mines where limestone is extracted, and coal mines where coal is extracted.

[0011] At the work site 1, the work machine 3 is in operation. The work machine 3 includes at least a first work machine 3A and a second work machine 3B. Note that the number of work machines 3 operating at the work site 1 is not limited to two. There may be one work machine 3 or any number of three or more work machines 3 operating at the work site 1. Examples of work machines 3 include bulldozers, shovels, wheel loaders, and motor graders. In this embodiment, the work machine 3 is a bulldozer. The work machine 3 performs excavation work, soil pushing work, and leveling work at the work site 1.

[0012] At work site 1, the aircraft 4 is in operation. The aircraft 4 is a mobile vehicle that can move around work site 1. The aircraft 4 is capable of flying above work site 1. The aircraft 4 is an unmanned aerial vehicle (UAV), such as a drone.

[0013] The management system 2 comprises a management device 9 and a communication system 10. The management device 9 includes a computer. The management device 9 is located outside the work machine 3 and the aircraft 4. The management device 9 is installed in the control facility 11 of the work site 1. The management device 9 manages the work site 1. The management device 9 manages at least the work machine 3 and the aircraft 4. The communication system 10 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN).

[0014] The work machine 3 includes a controller 12 and a wireless communication device 10A. The controller 12 includes a computer. The wireless communication device 10A is connected to the controller 12.

[0015] The aircraft 4 includes a controller 16 and a radio communication device 10B. The controller 16 includes a computer. The radio communication device 10B is connected to the controller 16.

[0016] The communication system 10 includes a wireless communication device 10A connected to the controller 12 of the work machine 3, a wireless communication device 10B connected to the controller 16 of the aircraft 4, and a wireless communication device 10C connected to the management device 9. The management device 9 and the controller 12 of the work machine 3 communicate wirelessly via the communication system 10. The management device 9 and the controller 16 of the aircraft 4 communicate wirelessly via the communication system 10.

[0017] In this embodiment, the work machine 3 is remotely controlled by a remote control system 13. At least a portion of the remote control system 13 is located in a remote control room 17. The control device 9 is located outside the work machine 3. The remote control room 17 is installed in a remote location away from the work site 1. The remote control system 13 comprises a remote control device 18, a display device 19, an input device 15, and a remote controller 20.

[0018] The remote control device 18 is located in the remote control room 17. The remote control device 18 is operated by an operator in the remote control room 17. When the remote control device 18 is operated, an operation signal is generated to operate the work machine 3. The operator can operate the remote control device 18 while seated in the control seat 21.

[0019] The display device 19 is located in the remote control room 17. The display device 19 displays captured images of the work site 1. The display device 19 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). The operator operates the remote control device 18 while checking the captured images of the work site 1 displayed on the display device 19. The work machine 3 is remotely controlled by the remote control device 18.

[0020] The input device 15 is located in the remote control room 17. The input device 15 is operated by an operator in the remote control room 17. Input data is generated when the input device 15 is operated. In this embodiment, the input device 15 includes a touch sensor located on the display screen of the display device 19. The display device 19 includes a touch panel. The input device 15 may be a computer keyboard, a mouse, or a voice input device.

[0021] The remote controller 20 is located in the remote control room 17. The remote controller 20 includes a computer. The remote controller 20 and the controller 12 of the first work machine 3A communicate via a communication system 22. The communication system 22 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN). Communication system 10 and communication system 22 may be separate communication systems. Communication system 10 and at least a portion of communication system 22 may be the same communication system.

[0022] In this embodiment, at least the first work machine 3A is remotely controlled by the remote control system 13. The second work machine 3B may or may not be remotely controlled. The second work machine 3B may be operated by an operator sitting in the cab of the second work machine 3B. The second work machine 3B may be autonomously controlled by the controller 12.

[0023] [Work Machine] Figure 2 is a schematic side view showing a work machine 3 according to an embodiment. As shown in Figure 2, the work machine 3 comprises a vehicle body 23, a traveling device 24, an excavating work machine 25, a ripper work machine 26, a three-dimensional sensor 27, a position sensor 28, an attitude sensor 29, and a camera 30. The vehicle body 23 supports an engine (not shown). The engine is the power source for the work machine 3. The traveling device 24 supports the vehicle body 23 and moves. The traveling device 24 has a pair of tracks 31. The work machine 3 moves as the tracks 31 rotate.

[0024] The excavation work machine 25 performs excavation, soil pushing, and leveling work on the target area. The excavation work machine 25 is attached to the vehicle body 23. At least a portion of the excavation work machine 25 is positioned in front of the vehicle body 23. The excavation work machine 25 has an excavation blade 32, a lift frame 33, a tilt cylinder 34, and a lift cylinder 35.

[0025] The drilling blade 32 is positioned in front of the vehicle body 23. The drilling blade 32 has a cutting edge 32A. The lift frame 33 supports the drilling blade 32. One end of the lift frame 33 is connected to the back of the drilling blade 32 via a rotating mechanism. The other end of the lift frame 33 is connected to the vehicle body 23 via a rotating mechanism. The other end of the lift frame 33 may also be connected to the traveling device 24 via a rotating mechanism.

[0026] The tilt cylinder 34 and the lift cylinder 35 each operate the drilling blade 32. The tilt cylinder 34 is driven to tilt the drilling blade 32. The lift cylinder 35 is driven to move the drilling blade 32 up and down. One end of the tilt cylinder 34 is connected to the back of the drilling blade 32 via a pivot mechanism. The other end of the tilt cylinder 34 is connected to the upper surface of the lift frame 33. As the tilt cylinder 34 extends and retracts, the tilt angle of the drilling blade 32 changes. One end of the lift cylinder 35 is connected to the lift frame 33 via a pivot mechanism. The other end of the lift cylinder 35 is connected to the vehicle body 23 via a pivot mechanism. As the lift cylinder 35 extends and retracts, the drilling blade 32 moves in the vertical direction.

[0027] The ripper implement 26 performs ripping operations, including cutting or crushing operations on the workpiece. The ripper implement 26 is mounted on the vehicle body 23. At least a portion of the ripper implement 26 is positioned behind the vehicle body 23. The ripper implement 26 has a shank 36, a ripper arm 37, a tilt cylinder 38, a lift cylinder 39, and a beam 40. The shank 36 is positioned behind the vehicle body 23. The shank 36 has a ripper point 36A. The ripper point 36A is provided at the tip of the shank 36. The ripper arm 37 supports the shank 36. The ripper arm 37 connects the vehicle body 23 and the shank 36. One end of the ripper arm 37 is connected to the rear of the vehicle body 23 via a pivot mechanism. The other end of the ripper arm 37 is connected to the beam 40. The beam 40 is rotatably connected to the ripper arm 37. The shank 36 is connected to the ripper arm 37 via the beam 40.

[0028] The tilt cylinder 38 and the lift cylinder 39 each operate the shank 36. The tilt cylinder 38 and the lift cylinder 39 are each connected to the vehicle body 23. The tilt cylinder 38 drives the shank 36 to tilt. The lift cylinder 39 drives the shank 36 to move up and down. One end of the tilt cylinder 38 is connected to the beam 40 via a pivot mechanism. The other end of the tilt cylinder 38 is connected to the rear of the vehicle body 23. As the tilt cylinder 38 extends and retracts, the tilt angle of the shank 36 changes. The tilt cylinder 38 moves the shank 36 in the front-rear direction. One end of the lift cylinder 39 is connected to the beam 40 via a pivot mechanism. The other end of the lift cylinder 39 is connected to the rear of the vehicle body 23. As the lift cylinder 39 extends and retracts, the shank 36 moves in the up-and-down direction. The lift cylinder 39 moves the shank 36 in the up-and-down direction.

[0029] The ripper work machine 26 pierces the workpiece with its ripper point 36A. As the traveling device 24 moves with the ripper point 36A embedded in the workpiece, the workpiece is cut or crushed. The shank 36 may move vertically and horizontally while the traveling device 24 is moving.

[0030] The 3D sensor 27 detects the 3D shape of the object to be detected. The 3D sensor 27 is positioned on the vehicle body 23. The 3D sensor 27 detects the 3D shape of the object to be detected without contact with it. The object to be detected by the 3D sensor 27 includes the work site 1 surrounding the work machine 3. The 3D sensor 27 detects the 3D shape of the work site 1 surrounding the work machine 3. The object to be detected by the 3D sensor 27 includes the terrain of the work site 1. The object to be detected by the 3D sensor 27 includes the ground of the work site 1 surrounding the work machine 3 and objects surrounding the work machine 3. The object to be detected by the 3D sensor 27 may include at least a part of the work machine 3. The 3D sensor 27 detects the distance to the surface of the object to be detected. The 3D sensor 27 detects the 3D shape of the surface of the object to be detected by detecting the relative distance to each of a plurality of detection points on the surface of the object to be detected. The 3D data showing the 3D shape of the object to be detected includes 3D point cloud data consisting of a plurality of detection points. The 3D data includes the relative distance and relative position between the 3D sensor 27 and each of the multiple detection points defined for the object to be detected. The 3D data also includes the height data for each of the multiple detection points. An example of the 3D sensor 27 is a laser sensor (LIDAR: Light Detection and Ranging) that detects the object by emitting laser light. The 3D sensor 27 may also be a 3D camera such as a stereo camera.

[0031] The position sensor 28 detects the position of the work machine 3. The position sensor 28 detects the position of the work machine 3 when at least the 3D sensor 27 has detected the 3D shape of the object to be detected. The position sensor 28 is located on the vehicle body 23. The position of the work machine 3 is detected using the Global Navigation Satellite System (GNSS). The Global Navigation Satellite System includes the Global Positioning System (GPS). The Global Navigation Satellite System detects the position in a global coordinate system defined by latitude, longitude, and altitude coordinate data. A global coordinate system is a coordinate system fixed to the Earth. The position sensor 28 includes a GNSS receiver located on the vehicle body 23. The position sensor 28 detects the position of the work machine 3 in the global coordinate system.

[0032] The attitude sensor 29 detects the attitude of the work machine 3. The attitude sensor 29 detects the attitude of the work machine 3 when at least the three-dimensional sensor 27 has detected the three-dimensional shape of the object to be detected. The attitude sensor 29 is positioned on the vehicle body 23. The attitude of the work machine 3 includes the tilt of the vehicle body 23. The attitude of the work machine 3 includes the tilt angle of the vehicle body 23 with respect to the horizontal plane. The attitude of the work machine 3 includes the orientation of the vehicle body 23. The orientation of the vehicle body 23 includes, for example, the orientation of the front end of the vehicle body 23. The orientation of the vehicle body 23 includes the bearing of the vehicle body 23. An inertial measuring unit (IMU) is exemplified as the attitude sensor 29. The attitude sensor 29 is capable of detecting the tilt angle of the vehicle body 23 with respect to the horizontal plane.

[0033] Camera 30 captures an image of the object to be imaged. Camera 30 is positioned on the vehicle body 23. The object to be imaged by camera 30 includes the work site 1 surrounding the work machine 3. The object to be imaged by camera 30 includes the ground of the work site 1 surrounding the work machine 3 and objects around the work machine 3. The object to be imaged by camera 30 includes at least the work site 1 in front of the work machine 3. In the example shown in Figure 2, camera 30 is positioned on the vehicle body 23 to capture the work site 1 in front of the work machine 3. Alternatively, camera 30 may be positioned on the vehicle body 23 to capture the work site 1 behind the work machine 3. Alternatively, camera 30 may be positioned on the vehicle body 23 to capture at least a part of the work machine 3. The image of the object to be imaged captured by camera 30 is displayed on the display device 19 in the remote control room 17. Camera 30 includes an RGB camera.

[0034] Camera 30 captures an image of the target when at least the 3D sensor 27 has detected the 3D shape of the target. The imaging range of camera 30 and at least a portion of the detection range of the 3D sensor 27 overlap. The image of camera 30 includes the target detected by the 3D sensor 27. Camera 30 captures an image of the target detected by the 3D sensor 27.

[0035] Figure 3 is a schematic plan view showing the work machine 3 according to the embodiment. As shown in Figure 3, the three-dimensional sensor 27 mounted on the work machine 3 has a detection range DR. The three-dimensional sensor 27 detects three-dimensional data of a target to be detected located within the detection range DR. In this embodiment, the three-dimensional sensor 27 includes a three-dimensional sensor 27F that detects three-dimensional data in front of the vehicle body 23 and a three-dimensional sensor 27B that detects three-dimensional data behind the vehicle body 23. The detection range DR of the three-dimensional sensor 27 includes the detection range DRF of the three-dimensional sensor 27F and the detection range DRB of the three-dimensional sensor 27B. At least a portion of the detection range DRF is defined in front of the excavation work machine 25. At least a portion of the detection range DRB is defined behind the ripper work machine 26.

[0036] [Remote Controller] Figure 4 is a hardware configuration diagram showing a remote controller 20 according to an embodiment. The remote controller 20 includes a computer. The remote controller 20 has a processor 44 such as a CPU (Central Processing Unit), a main memory 45 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 46, an input / output interface 47 including input / output circuits, and a communication interface 48 including communication circuits. The functions of the remote controller 20 are stored in the storage 46 as a computer program 49. The processor 44 reads the computer program 49 from the storage 46, loads it into the main memory 45, and executes processing according to the computer program 49. The computer program 49 may be distributed to the remote controller 20 via a network.

[0037] Similar to the remote controller 20, the management device 9, controller 12, and controller 16 each include a computer. Each of the management device 9, controller 12, and controller 16 also has a processor, main memory, storage for storing computer programs, an input / output interface, and a communication interface.

[0038] Figure 5 is a functional block diagram showing the management system 2 according to the embodiment. Each of the controller 12, the management device 9, and the remote controller 20 has a plurality of functional units. The functions of the functional units of the controller 12 are performed by the processor of the controller 12. The functions of the functional units of the management device 9 are performed by the processor of the management device 9. The functions of the functional units of the remote controller 20 are performed by the processor 44 of the remote controller 20.

[0039] The functional unit of controller 12 includes a detection data transmission unit 41, an operation signal acquisition unit 42, and a control unit 43. The functional unit of management device 9 includes a work site image generation unit 60 and a detection data transfer unit 61. The functional unit of remote controller 20 includes a detection data acquisition unit 50, an input data acquisition unit 51, a 3D image generation unit 52, a model generation unit 53, a virtual wall setting unit 54, an obstacle detection unit 55, a display data generation unit 56, and an operation signal transmission unit 57. The remote controller 20 also has a work site image storage unit 58. The function of the work site image storage unit 58 is performed by the storage unit 46 of the remote controller 20.

[0040] The controller 12 of the first work machine 3A is connected to the 3D sensor 27, the position sensor 28, the attitude sensor 29, and the camera 30. The detection data transmission unit 41 of the controller 12 transmits to the remote controller 20 3D data showing the 3D shape of the work site 1 detected by the 3D sensor 27, position data showing the position of the first work machine 3A detected by the position sensor 28, attitude data showing the attitude of the first work machine 3A detected by the attitude sensor 29, and the image of the work site 1 captured by the camera 30. The remote controller 20 displays the image of the work site 1 captured by the camera 30 on the display device 19.

[0041] The remote controller 20 transmits the operation signal generated in the remote control device 18 to the first work machine 3A. The operation signal acquisition unit 42 receives the operation signal from the remote control device 18. The control unit 43 outputs a control command to operate at least a part of the work machine 3 based on the operation signal from the remote control device 18. The control unit 43 outputs a control command to control the operation of at least a part of the excavation work machine 25, the ripper work machine 26, and the travel device 24 based on the operation signal from the remote control device 18. Controlling the operation of the excavation work machine 25 includes controlling the operation of at least one of the tilt cylinder 34 and the lift cylinder 35. Controlling the operation of the ripper work machine 26 includes controlling the operation of at least one of the tilt cylinder 38 and the lift cylinder 39.

[0042] The flying object 4 includes a three-dimensional sensor 5, a position sensor 6, an attitude sensor 7, and a camera 8. The three-dimensional sensor 5 detects a three-dimensional shape of the work site 1. A laser sensor (LIDAR: Light Detection and Ranging) is exemplified as the three-dimensional sensor 5. The three-dimensional sensor 5 may also be a three-dimensional camera such as a stereo camera or an RGB-D camera. The position sensor 6 detects the position of the flying object 4. The position sensor 6 includes a GNSS receiver. The position sensor 6 detects the position of the flying object 4 in a global coordinate system. The attitude sensor 7 detects the attitude of the flying object 4. The attitude of the flying object 4 includes the inclination angle of the flying object 4 with respect to a horizontal plane. The attitude of the flying object 4 includes the azimuth of the flying object 4. An inertial measurement unit (IMU: Inertial Measurement Unit) is exemplified as the attitude sensor 7. The camera 8 captures an image of the work site 1. The camera 8 includes an RGB camera.

[0043] FIG. 6 is a diagram for explaining the operation of the flying object 4 according to the embodiment. For example, before the work machine 3 starts work, the flying object 4 flies over the work site 1. While the flying object 4 flies over the work site 1, the three-dimensional sensor 5 of the flying object 4 detects the three-dimensional shape of the work site 1. A wide range of the work site 1 is detected by the three-dimensional sensor 5 of the flying object 4. For example, the three-dimensional shape of the entire work site 1 is detected by the three-dimensional sensor 5. The three-dimensional shape of the work site 1 includes the three-dimensional shape of the terrain of the work site 1.

[0044] In the following description, three-dimensional data representing the three-dimensional shape of the work site 1 detected by the three-dimensional sensor 5 of the flying object 4 is appropriately referred to as map data. The range of the map data is wider than the detection range DR of the three-dimensional sensor 27 of the work machine 3. The map data may be, for example, a three-dimensional shape of the entire work site 1.

[0045] The position sensor 6 detects the position of the aircraft 4 when at least the 3D sensor 5 has detected the 3D shape of the work site 1. The attitude sensor 7 detects the attitude of the aircraft 4 when at least the 3D sensor 5 has detected the 3D shape of the work site 1. The camera 8 images the work site 1 when at least the 3D sensor 5 has detected the 3D shape of the work site 1. The imaging range of the camera 8 and at least a portion of the detection range of the 3D sensor 5 overlap. The camera 8 images the terrain of the work site 1 that is within the detection range of the 3D sensor 5.

[0046] The controller 16 transmits to the management device 9 map data showing the three-dimensional shape of the work site 1 detected by the three-dimensional sensor 5, position data showing the position of the aircraft 4 detected by the position sensor 6, attitude data showing the attitude of the aircraft 4 detected by the attitude sensor 7, and images of the work site 1 captured by the camera 8.

[0047] The work site image generation unit 60 of the management device 9 generates a work site image 70 (see Figure 7) showing a three-dimensional image of the work site 1 based on the map data of the work site 1 detected by the three-dimensional sensor 5, the position data of the aircraft 4 detected by the position sensor 6, the attitude data of the aircraft 4 detected by the attitude sensor 7, and the image of the work site 1 captured by the camera 8. The work site image generation unit 60 generates map data in a global coordinate system based on the position data and attitude data of the aircraft 4. The map data includes three-dimensional point cloud data consisting of multiple detection points. The work site image generation unit 60 calculates the position of each of the multiple detection points in the global coordinate system based on the position data and attitude data of the aircraft 4.

[0048] The worksite image generation unit 60 adds color to map data based on the captured image of the worksite 1 captured by the camera 8. When the camera 8 is an RGB camera, the worksite image generation unit 60 can add color corresponding to the color of the worksite 1 to the map data based on the captured image of the worksite 1 captured by the camera 8. In the embodiment, the colored map data (three-dimensional point cloud data) is the worksite image 70. The worksite image 70 generated by the worksite image generation unit 60 is transmitted to the remote controller 20.

[0049] Although not illustrated in FIG. 5, similar to the first work machine 3A, the second work machine 3B includes a three-dimensional sensor 27, a position sensor 28, an orientation sensor 29, and a camera 30. The second work machine 3B may or may not be remotely operated. The controller 12 of the second work machine 3B transmits, to the management device 9, three-dimensional data indicating the three-dimensional shape of the worksite 1 around the second work machine 3B detected by the three-dimensional sensor 27, position data indicating the position of the second work machine 3B detected by the position sensor 28, orientation data indicating the orientation of the second work machine 3B detected by the orientation sensor 29, and the captured image of the worksite 1 around the second work machine 3B captured by the camera 30, respectively.

[0050] The detection data transfer unit 61 of the management device 9 transmits, to the remote controller 20, three-dimensional data indicating the three-dimensional shape of the worksite 1 around the second work machine 3B detected by the three-dimensional sensor 27, position data indicating the position of the second work machine 3B detected by the position sensor 28, orientation data indicating the orientation of the second work machine 3B detected by the orientation sensor 29, and the captured image of the worksite 1 around the second work machine 3B captured by the camera 30.

[0051] The detection data acquisition unit 50 of the remote controller 20 acquires three-dimensional data showing the three-dimensional shape of the work site 1 surrounding the first work machine 3A, detected by the three-dimensional sensor 27 of the first work machine 3A; position data showing the position of the first work machine 3A, detected by the position sensor 28 of the first work machine 3A; posture data showing the posture of the first work machine 3A, detected by the posture sensor 29 of the first work machine 3A; and an image of the work site 1 surrounding the first work machine 3A, captured by the camera 30 of the first work machine 3A. The position data of the first work machine 3A indicates the position of the first work machine 3A when the three-dimensional sensor 27 has detected the three-dimensional shape of the work site 1 surrounding the first work machine 3A. The posture data of the first work machine 3A indicates the posture of the first work machine 3A when the three-dimensional sensor 27 has detected the three-dimensional shape of the work site 1 surrounding the first work machine 3A. The captured images of the work site 1 surrounding the first work machine 3A include images taken when the 3D sensor 27 has detected the 3D shape of the work site 1 surrounding the first work machine 3A. The input data acquisition unit 51 acquires input data from the input device 15.

[0052] The 3D image generation unit 52 generates a 3D image 71 (see Figure 7) of the work site 1 surrounding the first work machine 3A based on the 3D data of the work site 1 surrounding the first work machine 3A acquired by the detection data acquisition unit 50. In this embodiment, the 3D image generation unit 52 generates a 3D image 71 of the work site 1 based on the 3D data of the work site 1 surrounding the first work machine 3A acquired by the detection data acquisition unit 50, the position data of the first work machine 3A, the posture data of the first work machine 3A, and the captured image of the work site 1.

[0053] The 3D image generation unit 52 generates 3D data in a global coordinate system based on the position data and orientation data of the first work machine 3A. The 3D data includes 3D point cloud data consisting of multiple detection points. The 3D image generation unit 52 calculates the position of each of the multiple detection points in the global coordinate system based on the position data and orientation data of the first work machine 3A.

[0054] The 3D image generation unit 52 adds color to the 3D data based on the image of the work site 1 surrounding the first work machine 3A captured by the camera 30. If the camera 30 is an RGB camera, the 3D image generation unit 52 can add colors to the 3D data that correspond to the colors of the work site 1 surrounding the first work machine 3A, based on the image of the work site 1 captured by the camera 30. In this embodiment, the colored 3D data (3D point cloud data) is the 3D image 71.

[0055] The model generation unit 53 generates a first model 300A (see Figure 7) that represents a model of the first work machine 3A. The first model 300A is a model that mimics the appearance of the first work machine 3A. The model generation unit 53 generates the first model 300A of the first work machine 3A based on at least the position data of the first work machine 3A. In this embodiment, the model generation unit 53 generates the first model 300A of the first work machine 3A based on the position data and the posture data of the first work machine 3A.

[0056] The virtual wall setting unit 54 sets a virtual wall at the work site 1 that restricts the entry of the first work machine 3A. The virtual wall restricts the operating range of the first work machine 3A. The virtual wall is set to surround the first work machine 3A. The virtual wall is set in the global coordinate system. The virtual wall is set to the south, north, east, west, above, and below the first work machine 3A. Note that the virtual wall may be set to at least one of the south, north, east, west, above, and below the first work machine 3A. Even if the first work machine 3A moves at the work site 1, the virtual wall does not move. The virtual wall is fixed at the work site 1. The area outside the virtual wall is an avoidance area where the entry of the first work machine 3A should be avoided. When the first working machine 3A approaches a virtual wall, the operation of at least one of the traveling device 24, the excavating machine 25, and the ripper machine 26 is restricted, or an alarm is output in the remote control room 17.

[0057] There is a possibility that obstacles exist at the work site 1 that could hinder the operation of the first work machine 3A. Examples of obstacles include structures located above the first work machine 3A, cliffs or holes in the work site 1, and buildings. The avoidance area is, for example, the area where the obstacles exist. If the first work machine 3A enters the avoidance area, the work efficiency of the first work machine 3A may decrease. By setting a virtual wall at the boundary between the work area where the first work machine 3A operates and the avoidance area, the first work machine 3A is prevented from crossing the virtual wall and entering the avoidance area. This prevents a decrease in the work efficiency of the first work machine 3A. An example of a virtual wall is disclosed in Japanese Patent Application Publication No. 2024-034408.

[0058] The obstacle detection unit 55 detects obstacles present in the work site 1 based on the three-dimensional data of the work site 1 surrounding the first work machine 3A detected by the three-dimensional sensor 27 of the first work machine 3A. Based on the three-dimensional data, the obstacle detection unit 55 sets the ground on which the first work machine 3A travels as the reference plane of the work site 1. The ground of the work site 1 on which the first work machine 3A travels is the ground in contact with the contact surface of the tracks 31 of the running gear 24 of the first work machine 3A. Based on the three-dimensional data, the obstacle detection unit 55 detects an object as an obstacle if it determines that an object exists whose protrusion from the reference plane of the work site 1 is greater than or equal to a predetermined height threshold.

[0059] The display data generation unit 56 generates display data to be displayed on the display device 19. The display data to be displayed on the display device 19 includes at least a three-dimensional image 71 of the work site 1 surrounding the first work machine 3A. In this embodiment, the display data generation unit 56 generates the display data so that the first model 300A of the first work machine 3A is displayed on the display device 19 together with the three-dimensional image 71. The display data generation unit 56 generates the display data so that the three-dimensional image 71 and the first model 300A are superimposed on a work site image 70 generated based on map data showing the three-dimensional shape of the work site 1.

[0060] When a virtual wall is set by the virtual wall setting unit 54, the display data generation unit 56 generates display data so that a virtual wall image 73 (see Figure 8) representing the virtual wall is displayed on the display device 19. When an obstacle is detected by the obstacle detection unit 55, the display data generation unit 56 generates display data so that an obstacle image 74 (see Figure 8) representing the obstacle is displayed on the display device 19.

[0061] The operation signal transmission unit 57 transmits an operation signal generated by the operation of the remote control device 18 to the first work machine 3A. The control unit 43 of the first work machine 3A outputs a control command to operate at least a part of the work machine 3 based on the operation signal from the remote control device 18.

[0062] The work site image storage unit 58 stores the work site image 70 generated by the work site image generation unit 60. The work site image 70 is pre-generated based on map data showing the three-dimensional shape of the work site 1 over a range wider than the detection range DR of the three-dimensional sensor 27 of the first work machine 3A.

[0063] [Display Device] Figure 7 shows an example of display data 100A displayed on the display device 19 according to the embodiment. The display data generation unit 56 generates display data 100A that includes a work site image 70, a three-dimensional image 71 of the surroundings of the first work machine 3A, and a first model 300A. The positions of the work site image 70, the three-dimensional image 71, and the first model 300A are each defined in a global coordinate system. The display data generation unit 56 causes the display data 100A to be displayed on the display device 19.

[0064] The work site image 70 is, for example, a three-dimensional image of the entire work site 1. The three-dimensional image 71 of the area around the first work machine 3A is displayed superimposed on the work site image 70. The three-dimensional image 71 is generated based on three-dimensional data detected by the three-dimensional sensor 27 of the first work machine 3A. There is a one-to-one correspondence between the range of the three-dimensional image 71 and the detection range DR of the three-dimensional sensor 27. The range of the three-dimensional image 71 is smaller than the range of the work site image 70. The three-dimensional image 71 is superimposed on a portion of the work site image 70.

[0065] The first model 300A is displayed superimposed on the work site image 70. The first model 300A is displayed superimposed on the three-dimensional image 71. However, the first model 300A does not necessarily have to be superimposed on the three-dimensional image 71.

[0066] The model generation unit 53 generates a first model 300A based on the position data of the first work machine 3A detected by the position sensor 28 and the attitude data of the first work machine 3A detected by the attitude sensor 29. In response to changes in the position of the first work machine 3A, the position of the first model 300A changes on the display screen of the display device 19. In response to changes in the tilt (angle of inclination relative to the horizontal plane) of the first work machine 3A, the tilt of the first model 300A changes on the display screen of the display device 19. In response to changes in the orientation (direction) of the first work machine 3A, the orientation of the first model 300A changes on the display screen of the display device 19.

[0067] As the first work machine 3A moves across the work site 1, the first model 300A moves on the display screen of the display device 19. As the first work machine 3A moves across the work site 1, the detection range DR of the three-dimensional sensor 27 moves. As the detection range DR of the three-dimensional sensor 27 moves, the three-dimensional image 71 moves on the display screen of the display device 19.

[0068] The work site image 70 is generated in advance before the work of the work machine 3 begins. As the work of the work machine 3 progresses, the terrain of the work site 1 may change. The 3D sensor 27 of the first work machine 3A can detect 3D data of the terrain of the work site 1 after the change. The 3D sensor 27 of the first work machine 3A can detect 3D data of the latest terrain of the work site 1. The display data generation unit 56 updates at least a portion of the work site image 70 with a 3D image 71 of the area around the first work machine 3A. The display data generation unit 56 overwrites at least a portion of the work site image 70 with a 3D image 71 of the area around the first work machine 3A. Based on the 3D data of the work site 1 around the first work machine 3A detected by the 3D sensor 27, the position data of the first work machine 3A, and the posture data of the first work machine 3A, the display data generation unit 56 can update at least a portion of the work site image 70 with a 3D image 71 of the area around the first work machine 3A.

[0069] In the example shown in Figure 7, the obstacle detection unit 55 determines whether or not a sloping cliff exists at the work site 1 based on map data detected by the 3D sensor 5 or 3D data detected by the 3D sensor 27. Based on the 3D data, the obstacle detection unit 55 sets the ground on which the first work machine 3A travels as the reference plane of the work site 1. The ground at the work site 1 on which the first work machine 3A travels is the ground in contact with the contact surface of the tracks 31 of the travel device 24 of the first work machine 3A. Based on the 3D data, if the obstacle detection unit 55 determines that there is ground where the amount of subsidence from the reference plane of the work site 1 is greater than or equal to a predetermined height threshold, it detects that ground as a sloping cliff. If it is determined that a sloping cliff exists, the display data generation unit 56 displays the detected points 72 (point cloud) indicating the sloping cliff in a different display format than the detected points (point cloud) indicating the reference plane. That is, the display data generation unit 56 highlights the sloping cliff present at the work site 1.

[0070] Figure 8 shows an example of display data 100B displayed in the display device 19 according to the embodiment. The operator can change the viewpoint of the first model 300A by operating the input device 15. The input data generated by operating the input device 15 is acquired by the input data acquisition unit 51. The display data generation unit 56 can convert the three-dimensional image 71 into a viewpoint-converted image viewed from the viewpoint specified by the input data. Along with the conversion of the viewpoint of the three-dimensional image 71, the viewpoints of the first model 300A and the work site image 70 are also converted. The display data 100A shown in Figure 7 includes a three-dimensional image 71 (first model 300A) viewed from a viewpoint defined to the front right. The display data 100B shown in Figure 8 includes a three-dimensional image 71 (first model 300A) viewed from a viewpoint defined to the front left.

[0071] The operator can use the input device 15 to move their viewpoint, for example, above the work site 1. By moving the viewpoint above the work site 1, the display device 19 displays a three-dimensional image 71 (overhead view) of the work site 1 as seen from above. The operator can use the input device 15 to move their viewpoint, for example, to the driver's cab of the first work machine 3A. By moving the viewpoint to the driver's cab of the first work machine 3A, the display device 19 displays a three-dimensional image 71 (operator image) of the work site 1 as seen from the driver's cab. The operator can use the input device 15 to move their viewpoint to any position.

[0072] When a virtual wall is set on the first work machine 3A by the virtual wall setting unit 54, the display data generation unit 56 generates display data 100B including a virtual wall image 73 that represents the virtual wall. The display data generation unit 56 generates the display data 100A such that the virtual wall image 73 surrounds the first model 300A. The virtual wall image 73 is a frame-shaped image that surrounds the first model 300A. The display data generation unit 56 generates the display data 100A such that the virtual wall image 73 is superimposed on the work site image 70. The virtual wall image 73 may also be displayed superimposed on the first model 300A. The virtual wall image 73 may also be displayed superimposed on the three-dimensional image 71.

[0073] When an obstacle is detected by the obstacle detection unit 55, the display data generation unit 56 generates display data 100B that includes an obstacle image 74 indicating the obstacle. The display data generation unit 56 generates the display data 100B so that the obstacle image 74 is superimposed on the work site image 70. In the example shown in Figure 8, the display data generation unit 56 generates display data that includes an enhancement image 75 that highlights the obstacle image 74. The enhancement image 75 is a frame-shaped image that surrounds the obstacle image 74.

[0074] When the second work machine 3B is in operation at the work site 1, the model generation unit 53 generates a second model 300B that represents a model of the second work machine 3B. The second model 300B is a model that mimics the appearance of the second work machine 3B. The model generation unit 53 generates the second model 300B of the second work machine 3B based at least on the position data of the second work machine 3B. The display data generation unit 56 generates display data 100B so that the second model 300B is superimposed on the work site image 70.

[0075] In this embodiment, the model generation unit 53 generates a second model 300B based on the position data of the second work machine 3B detected by the position sensor 28 of the second work machine 3B and the attitude data of the second work machine 3B detected by the attitude sensor 29 of the second work machine 3B. The position of the second model 300B changes on the display screen of the display device 19 in response to changes in the position of the second work machine 3B. The tilt of the second model 300B changes on the display screen of the display device 19 in response to changes in the tilt (angle of inclination with respect to the horizontal plane) of the second work machine 3B. The orientation of the second model 300B changes on the display screen of the display device 19 in response to changes in the orientation (direction) of the second work machine 3B.

[0076] As the second work machine 3B moves across the work site 1, the second model 300B moves on the display screen of the display device 19. As the second work machine 3B moves across the work site 1, the detection range DR of the 3D sensor 27 of the second work machine 3B moves. As the detection range DR of the 3D sensor 27 moves, the 3D image 71 generated based on the 3D data detected by the 3D sensor 27 of the second work machine 3B moves on the display screen of the display device 19.

[0077] As the work of the second work machine 3B progresses, the terrain of the work site 1 may change. The 3D sensor 27 of the second work machine 3B can detect 3D data of the terrain of the work site 1 after the change. The 3D sensor 27 of the second work machine 3B can detect the latest 3D data of the terrain of the work site 1. The display data generation unit 56 updates at least a portion of the work site image 70 with a 3D image 71 of the area around the second work machine 3B. The display data generation unit 56 overwrites at least a portion of the work site image 70 with a 3D image 71 of the area around the second work machine 3B. Based on the 3D data of the work site 1 around the second work machine 3B detected by the 3D sensor 27 of the second work machine 3B and the position data of the second work machine 3B, the display data generation unit 56 can update at least a portion of the work site image 70 with a 3D image 71 of the area around the second work machine 3B.

[0078] [Display Method] Figure 9 is a flowchart showing a method for displaying display data according to the embodiment. In the following description, the display data generation unit 56 generates display data such that the image of the work site 1 captured by the camera 30 of the first work machine 3A is displayed on the display device 19 together with a three-dimensional image 71 of the area around the first work machine 3A at the work site 1.

[0079] The detection data acquisition unit 50 of the remote controller 20 acquires three-dimensional data of the surrounding area of ​​the first work machine 3A detected by the three-dimensional sensor 27 of the first work machine 3A, position data of the first work machine 3A detected by the position sensor 28 of the first work machine 3A, and posture data of the first work machine 3A detected by the posture sensor 29 of the first work machine 3A (step S1). The three-dimensional image generation unit 52 generates a three-dimensional image 71 of the work site 1 surrounding the first work machine 3A based on the three-dimensional data of the surrounding area of ​​the first work machine 3A, the position data of the first work machine 3A, and the posture data of the first work machine 3A. The model generation unit 53 generates a first model 300A of the first work machine 3A based on the position data of the first work machine 3A and the posture data of the first work machine 3A (step S2). The display data generation unit 56 generates first display data including a three-dimensional image 71 of the work site 1 surrounding the first work machine 3A and a first model 300A of the first work machine 3A (step S3).

[0080] The detection data acquisition unit 50 acquires an image of the work site 1 captured by the camera 30 of the first work machine 3A (step S4). The display data generation unit 56 generates second display data including the image of the work site 1 (step S5).

[0081] The display data generation unit 56 causes the first display data and the second display data to be displayed on the display device 19 (step S6).

[0082] Figure 10 shows an example of a display device 19 according to the embodiment. The display data generation unit 56 sets a plurality of divided display areas on the display device 19. As shown in Figure 10, the display device 19 is set to have a first display area 19A, a second display area 19B, a third display area 19C, and a fourth display area 19D as a plurality of divided display areas. The first display area 19A displays an image of the work site 1 in front of the work machine 3. In the example shown in Figure 10, the image displayed in the first display area 19A includes an image of the drilling blade 32. The second display area 19B displays an image of the work site 1 behind the work machine 3. In the example shown in Figure 10, the image displayed in the second display area 19B includes an image of the shank 36. The third display area 19C displays an image showing at least a part of the work machine 3. In the example shown in Figure 10, the image displayed in the third display area 19C includes an image of the ripper point 36A. The fourth display area 19D displays display data 100A, which includes a work site image 70, a three-dimensional image 71, and a first model 300A, as described with reference to Figure 7. The first display data generated in step S3 of Figure 9 includes the display data 100A displayed in the fourth display area 19D. The second display data generated in step S5 of Figure 9 includes captured images displayed in the first display area 19A, the second display area 19B, and the third display area 19C, respectively. The first display data including the three-dimensional image 71 and the second display data including the captured images are displayed in parallel on the display device 19. The first display data and the second display data are displayed simultaneously on the display device 19.

[0083] As described above, the operator can change the viewpoint of the first model 300A by operating the input device 15. The display data generation unit 56 can convert the three-dimensional image 71 into a viewpoint-converted image A, which is viewed from a viewpoint in front of the work machine 3. The display data generation unit 56 can also convert the three-dimensional image 71 into a viewpoint-converted image B, which is viewed from a viewpoint behind the work machine 3. The display data generation unit 56 can also convert the three-dimensional image 71 into a viewpoint-converted image C, which is viewed from a viewpoint that includes the ripper point 36A. The display data generation unit 56 may display the viewpoint-converted image A in the first display area 19A instead of the captured image of the work site 1 in front of the work machine 3. The display data generation unit 56 may display the viewpoint-converted image B in the second display area 19B instead of the captured image of the work site 1 behind the work machine 3. The display data generation unit 56 may display the viewpoint-converted image C in the third display area 19C instead of the captured image of the ripper point 36A. The display data generation unit 56 may generate a three-dimensional image 71 of a predetermined desired viewpoint without changing the viewpoint of the first model 300A based on the operation of the operator's input device 15.

[0084] Furthermore, the display data generation unit 56 causes the icon 131, the tilt indicator 132, and the inclinometer 133 to be displayed on a part of the display device 19. The tilt indicator 132 and the inclinometer 133 display the pitch angle and roll angle of the first working machine 3A with respect to the horizontal plane.

[0085] [Effects] As described above, in this embodiment, the remote controller 20 includes a detection data acquisition unit 50 that acquires three-dimensional data showing the three-dimensional shape of the work site 1 detected by a three-dimensional sensor 27 of the first work machine 3A operating at the work site 1, a three-dimensional image generation unit 52 that generates a three-dimensional image 71 of the work site 1 based on the three-dimensional data, and a display data generation unit 56 that generates display data 100A (100B) including the three-dimensional image 71 for display on a display device 19 located outside the first work machine 3A.

[0086] In this embodiment, a three-dimensional image 71 of the work site 1 surrounding the first work machine 3A is displayed on the display device 19, allowing the operator in the remote control room 17 to properly understand the situation at the work site 1. As a result, a decrease in the work efficiency of the work machine 3 is suppressed.

[0087] The display data generation unit 56 generates display data 100A (100B) so that the first model 300A of the first work machine 3A is displayed on the display device 19 together with the three-dimensional image 71. This allows the operator to understand the position of the first work machine 3A at the work site 1.

[0088] The display data generation unit 56 generates display data 100A (100B) such that the three-dimensional image 71 and the first model 300A are superimposed on the work site image 70, which is generated based on map data showing the three-dimensional shape of the work site 1. This allows the operator to understand the location of the detection range DR of the three-dimensional sensor 27 and the location of the first work machine 3A within the entire work site 1.

[0089] When a virtual wall is set, the display data generation unit 56 generates display data 100B so that a virtual wall image 73 representing the virtual wall is displayed on the display device 19. This allows the operator to understand the position and size of the virtual wall at the work site 1.

[0090] When an obstacle is detected, the display data generation unit 56 generates display data 100B so that an obstacle image 74 indicating the obstacle is displayed on the display device 19. This allows the operator to understand the location and size of the obstacle at the work site 1.

[0091] The display data generation unit 56 updates at least a portion of the work site image 70 into a three-dimensional image 71. This allows the operator to recognize the latest terrain of the work site 1.

[0092] When the second work machine 3B is in operation at the work site 1, the display data generation unit 56 generates display data 100B so that the second model 300B of the second work machine 3B is superimposed on the work site image 70. This allows the operator remotely controlling the first work machine 3A to recognize the position of the second work machine 3B at the work site 1.

[0093] The display data generation unit 56 converts the three-dimensional image 71 into a viewpoint-transformed image viewed from a viewpoint specified by the input data. Images of the work site 1 viewed from various viewpoints are displayed on the display device 19, so the occurrence of blind spots is suppressed, and the operator in the remote control room 17 can properly grasp the situation of the work site 1.

[0094] As explained with reference to Figure 10, the first display data, which includes the three-dimensional image 71, and the second display data, which includes the captured image (monocular camera image) captured by the camera 30, are displayed in parallel on the display device 19. This allows the operator in the remote control room 17 to properly grasp the situation at the work site 1.

[0095] [Other Embodiments] In the embodiments described above, the first work machine 3A and the second work machine 3B are assumed to be of the same type (bulldozer). The first work machine 3A and the second work machine 3B may be of different types. For example, if the first work machine 3A is a bulldozer, the second work machine 3B may be other work machines such as an excavator, a wheel loader, and a motor grader.

[0096] In the above-described embodiment, at least a portion of the functional parts of the remote controller 20 may be provided in the management device 9 or the controller 12. At least a portion of the functional parts of the management device 9 may be provided in the controller 12 or the remote controller 20. At least a portion of the functional parts of the controller 12 may be provided in the remote controller 20 or the management device 9. For example, the work site image generation unit 60 may be provided in the remote controller 20 or the controller 12. For example, the 3D image generation unit 52 and the model generation unit 53 may be provided in the controller 12. In this case, the controller 12 may transmit the generated 3D image 71 of the work site 1 and the model of the work machine 3 to the remote controller 20. Alternatively, the 3D image generation unit 52 and the model generation unit 53 may be provided in the management device 9.

[0097] In the embodiments described above, each of the multiple functional units of the remote controller 20 may be configured by a separate computer (hardware). Each of the functional units of the management device 9 may be configured by a separate computer (hardware). Each of the multiple functional units of the controller 12 may be configured by a separate computer (hardware). The multiple functional units of the remote controller 20, the multiple functional units of the management device 9, and the multiple functional units of the controller 12 may all be configured by a single computer (hardware).

[0098] 1...Work site, 2...Management system, 3...Work machine (bulldozer), 3A...First work machine, 3B...Second work machine, 4...Aircraft, 5...3D sensor, 6...Position sensor, 7...Attitude sensor, 8...Camera, 9...Management device, 10...Communication system, 10A...Wireless communication device, 10B...Wireless communication device, 10C...Wireless communication device, 11...Control facility, 12...Controller, 13...Remote control system, 15...Input device, 16...Controller, 17...Remote control room, 18...Remote control device, 19...Display device, 19A...First display Area, 19B...Second display area, 19C...Third display area, 19D...Fourth display area, 20...Remote controller, 21...Driver's seat, 22...Communication system, 23...Vehicle body, 24...Running gear, 25...Excavator, 26...Ripper, 27...3D sensor, 27F...3D sensor, 27B...3D sensor, 28...Position sensor, 29...Attitude sensor, 30...Camera, 31...Track, 32...Excavator blade, 32A...Cutting edge, 33...Lift frame, 34...Tilt cylinder, 35...Lift cylinder, 36...Sha 36A...Ripper point, 37...Ripper arm, 38...Tilt cylinder, 39...Lift cylinder, 40...Beam, 41...Detection data transmission unit, 42...Operation signal acquisition unit, 43...Control unit, 44...Processor, 45...Main memory, 46...Storage, 47...Input / output interface, 48...Communication interface, 49...Computer program, 50...Detection data acquisition unit, 51...Input data acquisition unit, 52...3D image generation unit, 53...Model generation unit, 54...Virtual wall setting unit, 55...Fault Object detection unit, 56... Display data generation unit, 57... Operation signal transmission unit, 58... Work site image storage unit, 60... Work site image generation unit, 61... Detection data transfer unit, 70... Work site image, 71... 3D image, 72... Detection point, 73... Virtual wall image, 74... Obstacle image, 75... Enhanced image, 100A... Display data, 100B... Display data, 131... Icon, 132... Tilt indicator, 133... Inclinometer, 300A... First model, 300B... Second model, DR... Detection range, DRF... Detection range, DRB... Detection range.

Claims

1. A remote control system for a work machine, comprising a processor, the processor acquiring three-dimensional data showing the three-dimensional shape of the work site detected by a three-dimensional sensor of a first work machine operating at the work site, generating a three-dimensional image of the work site based on the three-dimensional data, and generating display data including the three-dimensional image for display on a display device located outside the first work machine.

2. The remote control system for a work machine according to claim 1, wherein the processor acquires position data indicating the position of the first work machine detected by a position sensor of the first work machine, generates a first model indicating a model of the first work machine based on the position data, and generates display data such that the first model is displayed on the display device together with the three-dimensional image.

3. The remote control system for a work machine according to claim 2, wherein the processor generates the display data such that the three-dimensional image and the first model are superimposed on a work site image generated based on map data showing the three-dimensional shape of the work site.

4. The remote control system for a work machine according to claim 3, wherein the processor sets a virtual wall at the work site that restricts the entry of the first work machine, and generates the display data such that a virtual wall image showing the virtual wall is displayed on the display device.

5. The remote control system for a work machine according to claim 4, wherein the processor generates the display data such that the virtual wall image is superimposed on the work site image.

6. The remote control system for a work machine according to claim 5, wherein the processor generates the display data such that the virtual wall image surrounds the first model.

7. The remote control system for a work machine according to claim 3, wherein the processor detects obstacles present at the work site based on the three-dimensional data and generates display data such that an obstacle image showing the obstacles is displayed on the display device.

8. The remote control system for a work machine according to claim 7, wherein the processor generates the display data such that the obstacle image is superimposed on the work site image.

9. The remote control system for a work machine according to claim 3, wherein the processor updates at least a portion of the work site image into the three-dimensional image.

10. The remote control system for a work machine according to claim 3, wherein the processor generates a second model representing a model of the second work machine based on position data of the second work machine, and generates display data such that the second model is superimposed on the work site image.

11. The remote control system for a work machine according to claim 1, wherein the processor acquires input data from an input device and converts the three-dimensional image into a viewpoint-transformed image viewed from a viewpoint specified by the input data.

12. The remote control system for a work machine according to claim 1, wherein the processor acquires an image of the work site captured by a camera on the first work machine, and generates display data so that the image is displayed on the display device together with the three-dimensional image.

13. A method for remotely operating a work machine, comprising: acquiring three-dimensional data showing the three-dimensional shape of a work site detected by a three-dimensional sensor of a first work machine operating at a work site; generating a three-dimensional image of the work site based on the three-dimensional data; and generating display data including the three-dimensional image for display on a display device located outside the first work machine.

14. A method for remotely operating a work machine according to claim 13, comprising: acquiring position data indicating the position of the first work machine detected by a position sensor of the first work machine; generating a first model indicating a model of the first work machine based on the position data; and generating display data such that the first model is displayed together with the three-dimensional image on the display device.

15. A method for remotely operating a work machine according to claim 14, comprising generating display data such that the three-dimensional image and the first model are superimposed on a work site image generated based on map data showing the three-dimensional shape of the work site.

16. A method for remotely operating a work machine according to claim 15, comprising: setting a virtual wall at the work site that restricts the entry of the first work machine; and generating display data such that a virtual wall image representing the virtual wall is displayed on the display device.

17. A method for remotely operating a work machine according to claim 16, comprising generating the display data such that the virtual wall image is superimposed on the work site image.

18. A method for remotely operating a work machine according to claim 17, comprising generating the display data such that the virtual wall image surrounds the first model.

19. A method for remotely operating a work machine according to claim 15, comprising: detecting obstacles present at the work site based on the three-dimensional data; and generating display data such that an obstacle image showing the obstacles is displayed on the display device.

20. A method for remotely operating a work machine according to claim 19, comprising generating the display data such that the obstacle image is superimposed on the work site image.

Images