Display system and display method

Abstract

This display system comprises a processor. The processor acquires three-dimensional data indicating a three-dimensional shape of a work site detected by a three-dimensional sensor provided in a work machine operating at the work site, acquires a captured image of the work site captured by a camera provided in the work machine, detects a specific portion of the work site on the basis of the three-dimensional data, generates a specific image indicating the specific portion, and generates display data to be displayed on a display device, the display data including the captured image and the specific image to be superimposed on the captured image.

Description

Display System and Display Method

[0001] The present disclosure relates to a display system and a display method.

[0002] In the technical field related to display systems, a display system as disclosed in Patent Document 1 is known.

[0003] Japanese Patent Application Laid-Open No. 2018-207244

[0004] When an operator of a work machine operates the work machine while checking an image of the work site displayed on a display device, if the situation of the work site cannot be appropriately grasped, the work efficiency may decrease.

[0005] An object of the present disclosure is to suppress a decrease in work efficiency.

[0006] According to the present disclosure, a display system including a processor is provided. The processor acquires 3D data indicating the 3D shape of the work site detected by a 3D sensor of a work machine operating at the work site, acquires a captured image of the work site captured by a camera of the work machine, detects a specific part of the work site based on the 3D data, generates a specific image indicating the specific part, and generates display data including the captured image for display on a display device and the specific image superimposed on the captured image.

[0007] According to the present disclosure, a decrease in work efficiency is suppressed.

[0008] FIG. 1 is a diagram schematically showing a remote operation system for a work site according to an embodiment. FIG. 2 is a side view schematically showing a work machine according to an embodiment. FIG. 3 is a plan view schematically showing a work machine according to an embodiment. FIG. 4 is a hardware configuration diagram showing a remote controller according to an embodiment. FIG. 5 is a functional block diagram showing a display system according to an embodiment. FIG. 6 is a diagram for explaining a specific part and a specific image according to an embodiment. FIG. 7 is a diagram for explaining a method of generating a route image according to an embodiment. FIG. 8 is a diagram for explaining a method of generating a mesh image according to an embodiment. FIG. 9 is a flowchart showing a method of displaying display data according to an embodiment. FIG. 10 is a diagram showing an example of a display device according to an embodiment.

[0009] The embodiments of this disclosure will be described below with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [Remote Control System] Figure 1 is a schematic diagram showing the remote control system 2 for the work machine 3 according to the embodiment. The work machine 3 operates at the work site 1. Examples of the work site 1 include mines or quarries. A mine refers to a place or business establishment where minerals are extracted. A quarry refers to a place or business establishment where stone materials are extracted. Examples of mines include metal mines where metals are extracted, non-metallic mines where limestone is extracted, and coal mines where coal is extracted.

[0011] The work machine 3 is remotely controlled by the remote control system 2. Examples of work machines 3 include bulldozers, shovels, wheel loaders, and motor graders. In one embodiment, the work machine 3 is a bulldozer. The work machine 3 performs excavation, soil pushing, and leveling work at the work site 1.

[0012] The work machine 3 includes a controller 12 and a wireless communication device 10A. The controller 12 includes a computer. The wireless communication device 10A is connected to the controller 12.

[0013] At least a portion of the remote control system 2 is located in the remote control room 17. The remote control room 17 is located outside the work machine 3. The remote control room 17 is installed in a remote location away from the work site 1. The remote control system 2 comprises a remote control device 18, a display device 19, and a remote controller 20.

[0014] The remote control device 18 is located in the remote control room 17. The remote control device 18 is operated by an operator in the remote control room 17. When the remote control device 18 is operated, an operation signal is generated to operate the work machine 3. The operator can operate the remote control device 18 while seated in the control seat 21.

[0015] The display device 19 is located in the remote control room 17. The display device 19 displays captured images of the work site 1. The display device 19 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). The operator operates the remote control device 18 while checking the captured images of the work site 1 displayed on the display device 19. The work machine 3 is remotely controlled by the remote control device 18.

[0016] The remote controller 20 is located in the remote control room 17. The remote controller 20 includes a computer. The remote controller 20 and the controller 12 of the work machine 3 communicate wirelessly via the communication system 10. The communication system 10 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN). The communication system 10 includes a wireless communication device 10A connected to the controller 12 of the work machine 3.

[0017] [Work Machine] Figure 2 is a schematic side view showing a work machine 3 according to an embodiment. As shown in Figure 2, the work machine 3 comprises a vehicle body 23, a traveling device 24, an excavating work machine 25, a ripper work machine 26, a three-dimensional sensor 27, a position sensor 28, an attitude sensor 29, and a camera 30. The vehicle body 23 supports an engine (not shown). The engine is the power source for the work machine 3. The traveling device 24 supports the vehicle body 23 and moves. The traveling device 24 has a pair of tracks 31. The tracks 31 rotate as the drive wheels 31D rotate. The work machine 3 moves as the tracks 31 rotate.

[0018] The excavation work machine 25 performs excavation, soil pushing, and leveling work on the target area. The excavation work machine 25 is attached to the vehicle body 23. At least a portion of the excavation work machine 25 is positioned in front of the vehicle body 23. The excavation work machine 25 has an excavation blade 32, a lift frame 33, a tilt cylinder 34, and a lift cylinder 35.

[0019] The drilling blade 32 is positioned in front of the vehicle body 23. The drilling blade 32 has a cutting edge 32A. The lift frame 33 supports the drilling blade 32. One end of the lift frame 33 is connected to the back of the drilling blade 32 via a rotating mechanism. The other end of the lift frame 33 is connected to the vehicle body 23 via a rotating mechanism. The other end of the lift frame 33 may also be connected to the traveling device 24 via a rotating mechanism.

[0020] The tilt cylinder 34 and the lift cylinder 35 each operate the drilling blade 32. The tilt cylinder 34 is driven to tilt the drilling blade 32. The lift cylinder 35 is driven to move the drilling blade 32 up and down. One end of the tilt cylinder 34 is connected to the back of the drilling blade 32 via a pivot mechanism. The other end of the tilt cylinder 34 is connected to the upper surface of the lift frame 33. As the tilt cylinder 34 extends and retracts, the tilt angle of the drilling blade 32 changes. One end of the lift cylinder 35 is connected to the lift frame 33 via a pivot mechanism. The other end of the lift cylinder 35 is connected to the vehicle body 23 via a pivot mechanism. As the lift cylinder 35 extends and retracts, the drilling blade 32 moves in the vertical direction.

[0021] The ripper implement 26 performs ripping operations, including cutting or crushing operations on the workpiece. The ripper implement 26 is mounted on the vehicle body 23. At least a portion of the ripper implement 26 is positioned behind the vehicle body 23. The ripper implement 26 has a shank 36, a ripper arm 37, a tilt cylinder 38, a lift cylinder 39, and a beam 40. The shank 36 is positioned behind the vehicle body 23. The shank 36 has a ripper point 36A. The ripper point 36A is provided at the tip of the shank 36. The ripper arm 37 supports the shank 36. The ripper arm 37 connects the vehicle body 23 and the shank 36. One end of the ripper arm 37 is connected to the rear of the vehicle body 23 via a pivot mechanism. The other end of the ripper arm 37 is connected to the beam 40. The beam 40 is rotatably connected to the ripper arm 37. The shank 36 is connected to the ripper arm 37 via the beam 40.

[0022] The tilt cylinder 38 and the lift cylinder 39 each operate the shank 36. The tilt cylinder 38 and the lift cylinder 39 are each connected to the vehicle body 23. The tilt cylinder 38 drives the shank 36 to tilt. The lift cylinder 39 drives the shank 36 to move up and down. One end of the tilt cylinder 38 is connected to the beam 40 via a pivot mechanism. The other end of the tilt cylinder 38 is connected to the rear of the vehicle body 23. As the tilt cylinder 38 extends and retracts, the tilt angle of the shank 36 changes. The tilt cylinder 38 moves the shank 36 in the front-rear direction. One end of the lift cylinder 39 is connected to the beam 40 via a pivot mechanism. The other end of the lift cylinder 39 is connected to the rear of the vehicle body 23. As the lift cylinder 39 extends and retracts, the shank 36 moves in the up-and-down direction. The lift cylinder 39 moves the shank 36 in the up-and-down direction.

[0023] The ripper work machine 26 pierces the workpiece with its ripper point 36A. As the traveling device 24 moves with the ripper point 36A embedded in the workpiece, the workpiece is cut or crushed. The shank 36 may move vertically and longitudinally while the traveling device 24 is moving.

[0024] The 3D sensor 27 detects the 3D shape of the object to be detected. The 3D sensor 27 is positioned on the vehicle body 23. The 3D sensor 27 detects the 3D shape of the object to be detected without contact with it. The object to be detected by the 3D sensor 27 includes the work site 1 surrounding the work machine 3. The 3D sensor 27 detects the 3D shape of the work site 1 surrounding the work machine 3. The object to be detected by the 3D sensor 27 includes the terrain of the work site 1. The object to be detected by the 3D sensor 27 includes the ground of the work site 1 surrounding the work machine 3 and objects surrounding the work machine 3. The object to be detected by the 3D sensor 27 may include at least a part of the work machine 3. The 3D sensor 27 detects the distance to the surface of the object to be detected. The 3D sensor 27 detects the 3D shape of the surface of the object to be detected by detecting the relative distance to each of a plurality of detection points on the surface of the object to be detected. The 3D data showing the 3D shape of the object to be detected includes 3D point cloud data consisting of a plurality of detection points. The 3D data includes the relative distance and relative position between the 3D sensor 27 and each of the multiple detection points defined for the object to be detected. The 3D data also includes the height data for each of the multiple detection points. An example of the 3D sensor 27 is a laser sensor (LIDAR: Light Detection and Ranging) that detects the object by emitting laser light. The 3D sensor 27 may also be a 3D camera such as a stereo camera or an RGB-D camera.

[0025] The position sensor 28 detects the position of the work machine 3. The position sensor 28 detects the position of the work machine 3 when at least the 3D sensor 27 has detected the 3D shape of the object to be detected. The position sensor 28 is located on the vehicle body 23. The position of the work machine 3 is detected using the Global Navigation Satellite System (GNSS). The Global Navigation Satellite System includes the Global Positioning System (GPS). The Global Navigation Satellite System detects the position in a global coordinate system defined by latitude, longitude, and altitude coordinate data. A global coordinate system is a coordinate system fixed to the Earth. The position sensor 28 includes a GNSS receiver located on the vehicle body 23. The position sensor 28 detects the position of the work machine 3 in the global coordinate system.

[0026] The attitude sensor 29 detects the attitude of the work machine 3. The attitude sensor 29 detects the attitude of the work machine 3 when at least the three-dimensional sensor 27 has detected the three-dimensional shape of the object to be detected. The attitude sensor 29 is positioned on the vehicle body 23. The attitude of the work machine 3 includes the tilt of the vehicle body 23. The attitude of the work machine 3 includes the tilt angle of the vehicle body 23 with respect to the horizontal plane. The attitude of the work machine 3 includes the orientation of the vehicle body 23. The orientation of the vehicle body 23 includes, for example, the orientation of the front end of the vehicle body 23. The orientation of the vehicle body 23 includes the bearing of the vehicle body 23. An inertial measuring unit (IMU) is exemplified as the attitude sensor 29. The attitude sensor 29 is capable of detecting the tilt angle of the vehicle body 23 with respect to the horizontal plane.

[0027] Camera 30 captures an image of the object to be imaged. Camera 30 is positioned on the vehicle body 23. The object to be imaged by camera 30 includes the work site 1 surrounding the work machine 3. The object to be imaged by camera 30 includes the ground of the work site 1 surrounding the work machine 3 and objects around the work machine 3. The object to be imaged by camera 30 includes at least the work site 1 in front of the work machine 3. In the example shown in Figure 2, camera 30 is positioned on the vehicle body 23 to capture the work site 1 in front of the work machine 3. Alternatively, camera 30 may be positioned on the vehicle body 23 to capture the work site 1 behind the work machine 3. Alternatively, camera 30 may be positioned on the vehicle body 23 to capture at least a part of the work machine 3. The image of the object to be imaged captured by camera 30 is displayed on the display device 19 in the remote control room 17. Camera 30 includes an RGB camera.

[0028] Camera 30 captures an image of the target when at least the 3D sensor 27 has detected the 3D shape of the target. The imaging range of camera 30 and at least a portion of the detection range of the 3D sensor 27 overlap. The image of camera 30 includes the target detected by the 3D sensor 27. Camera 30 captures an image of the target detected by the 3D sensor 27.

[0029] Figure 3 is a schematic plan view showing the work machine 3 according to the embodiment. As shown in Figure 3, the three-dimensional sensor 27 mounted on the work machine 3 has a detection range DR. The three-dimensional sensor 27 detects three-dimensional data of a target to be detected located within the detection range DR. In this embodiment, the three-dimensional sensor 27 includes a three-dimensional sensor 27F that detects three-dimensional data in front of the vehicle body 23 and a three-dimensional sensor 27B that detects three-dimensional data behind the vehicle body 23. The detection range DR of the three-dimensional sensor 27 includes the detection range DRF of the three-dimensional sensor 27F and the detection range DRB of the three-dimensional sensor 27B. At least a portion of the detection range DRF is defined in front of the excavation work machine 25. At least a portion of the detection range DRB is defined behind the ripper work machine 26.

[0030] [Remote Controller] Figure 4 is a hardware configuration diagram showing a remote controller 20 according to an embodiment. The remote controller 20 includes a computer. The remote controller 20 has a processor 44 such as a CPU (Central Processing Unit), a main memory 45 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 46, an input / output interface 47 including input / output circuits, and a communication interface 48 including communication circuits. The functions of the remote controller 20 are stored in the storage 46 as a computer program 49. The processor 44 reads the computer program 49 from the storage 46, loads it into the main memory 45, and executes processing according to the computer program 49. The computer program 49 may be distributed to the remote controller 20 via a network.

[0031] Similar to the remote controller 20, the controller 12 also includes a computer. The controller 12 also has a processor, main memory, storage for storing computer programs, an input / output interface, and a communication interface.

[0032] Figure 5 is a functional block diagram showing a display system 4 according to an embodiment. The display system 4 displays at least captured images of the work site 1 on the display device 19. Each of the controller 12 and the remote controller 20 has a plurality of functional units. The functions of the functional units of the controller 12 are performed by the processor of the controller 12. The functions of the functional units of the remote controller 20 are performed by the processor 44 of the remote controller 20.

[0033] The functional unit of controller 12 includes a detection data transmission unit 41, an operation signal acquisition unit 42, and a control unit 43. The functional unit of remote controller 20 includes a detection data acquisition unit 50, a detection unit 51, a specific image generation unit 52, and a display data generation unit 53.

[0034] The controller 12 is connected to the 3D sensor 27, the position sensor 28, the attitude sensor 29, and the camera 30. The detection data transmission unit 41 transmits to the remote controller 20 3D data showing the 3D shape of the work site 1 detected by the 3D sensor 27, position data showing the position of the work machine 3 detected by the position sensor 28, attitude data showing the attitude of the work machine 3 detected by the attitude sensor 29, and the image of the work site 1 captured by the camera 30. The remote controller 20 displays the image of the work site 1 captured by the camera 30 on the display device 19.

[0035] The remote controller 20 transmits an operation signal generated in the remote control device 18 to the work machine 3. The operation signal acquisition unit 42 receives the operation signal from the remote control device 18. The control unit 43 outputs a control command to operate at least a part of the work machine 3 based on the operation signal from the remote control device 18. The control unit 43 outputs a control command to control the operation of at least a part of the excavation work machine 25, the ripper work machine 26, and the travel device 24 based on the operation signal from the remote control device 18. Controlling the operation of the excavation work machine 25 includes controlling the operation of at least one of the tilt cylinder 34 and the lift cylinder 35. Controlling the operation of the ripper work machine 26 includes controlling the operation of at least one of the tilt cylinder 38 and the lift cylinder 39.

[0036] The detection data acquisition unit 50 acquires three-dimensional data showing the three-dimensional shape of the work site 1 surrounding the work machine 3, detected by the three-dimensional sensor 27 of the work machine 3; position data showing the position of the work machine 3, detected by the position sensor 28 of the work machine 3; posture data showing the posture of the work machine 3, detected by the posture sensor 29 of the work machine 3; and images of the work site 1 surrounding the work machine 3, captured by the camera 30 of the work machine 3. The position data of the work machine 3 indicates the position of the work machine 3 when the three-dimensional sensor 27 has detected the three-dimensional shape of the work site 1 surrounding the work machine 3. The posture data of the work machine 3 indicates the posture of the work machine 3 when the three-dimensional sensor 27 has detected the three-dimensional shape of the work site 1 surrounding the work machine 3. The images of the work site 1 surrounding the work machine 3 include images captured when the three-dimensional sensor 27 has detected the three-dimensional shape of the work site 1 surrounding the work machine 3.

[0037] The detection unit 51 detects specific parts present at the work site 1 based on the three-dimensional data acquired by the detection data acquisition unit 50. The detection unit 51 detects specific parts present near the path of the work machine 3. The detection unit 51 detects specific parts present in the direction of travel of the work machine 3. Specific parts include areas requiring attention from the operator. Specific parts include cliffs and obstacles present at the work site 1. Cliffs include downward and upward cliffs present near the path of or in the direction of travel of the work machine 3. Obstacles include protrusions present near the path of or in the direction of travel of the work machine 3.

[0038] The work machine 3 travels on the ground of the work site 1. During the travel of the work machine 3, the contact surface of the tracks 31 of the travel device 24 comes into contact with the ground of the work site 1. Based on the 3D data of the work site 1 surrounding the work machine 3 detected by the 3D sensor 27 of the work machine 3, the detection unit 51 sets the ground on which the work machine 3 travels as the reference plane of the work site 1. The ground of the work site 1 on which the work machine 3 travels is the ground in contact with the contact surface of the tracks 31 of the travel device 24. If the detection unit 51 determines, based on the 3D data, that there is a part of the work site 1 where the amount of subsidence from the reference plane exceeds a predetermined subsidence threshold, it detects that part as a downhill cliff. If the detection unit 51 determines, based on the 3D data, that there is a part of the work site 1 where the amount of protrusion from the reference plane exceeds a predetermined protrusion threshold, it detects that part as an obstacle or an uphill cliff.

[0039] The specific image generation unit 52 generates a specific image showing the specific part detected by the detection unit 51. If the specific part is a cliff, the specific image includes an enhanced image that highlights the boundary between the ground on which the work machine 3 is traveling and the cliff. If the specific part is an obstacle, the specific image includes an enhanced image that highlights the obstacle.

[0040] Figure 6 is a diagram illustrating specific parts and images according to the embodiment. In the example shown in Figure 6, the terrain 60 of the work site 1 includes the ground 61 and a sloping cliff 62. The work machine 3 travels on the ground 61 of the work site 1. In the example shown in Figure 6, the work machine 3 moves forward on the ground 61. The ground 61 extends in the direction of travel of the work machine 3. The sloping cliff 62 exists on both sides of the ground 61. The ground 61 and the sloping cliff 62 are adjacent in a direction perpendicular to the direction of travel of the work machine 3. Also, in the example shown in Figure 6, there is an obstacle 65 in the direction of travel of the work machine 3. The obstacle 65 is located on the ground 61.

[0041] The 3D sensor 27 detects 3D data of the work site 1 while the work machine 3 is in motion. The camera 30 captures images of the work site 1 while the work machine 3 is in motion. The detection data acquisition unit 50 acquires at least the 3D data of the work site 1 and the captured images of the work site 1 from the controller 12 while the work machine 3 is in motion.

[0042] During the travel of the working machine 3, the ground contact surface of the crawler 31 of the traveling device 24 contacts the ground 61 of the work site 1. The detection unit 51 sets the ground 61 on which the working machine 3 travels as the reference plane of the work site 1 based on the three-dimensional data of the work site 1 detected by the three-dimensional sensor 27. Note that the ground contact surface of the crawler 31 may be set as the reference plane. When the detection unit 51 determines, based on the three-dimensional data, that there is a portion where the subsidence amount from the reference plane (ground 61) of the work site 1 is equal to or greater than a predetermined subsidence threshold value, the detection unit 51 detects that portion as a cliff 62. When the detection unit 51 determines, based on the three-dimensional data, that there is a portion where the protrusion amount from the reference plane (ground 61) of the work site 1 is equal to or greater than a predetermined protrusion threshold value, the detection unit 51 detects that portion as an obstacle 65.

[0043] The specific image generation unit 52 generates a specific image 70 indicating a specific portion. In the example shown in FIG. 6, the specific image 70 includes a first specific image 70A indicating the cliff 62 and a second specific image 70B indicating the obstacle 65. The first specific image 70A is an emphasized image that emphasizes the cliff 62. In the embodiment, the first specific image 70A is an emphasized image that emphasizes the boundary 63 between the ground 61 and the cliff 62. The second specific image 70B is an emphasized image that emphasizes the obstacle 65.

[0044] The first specific image 70A is generated so as to overlap the boundary 63 between the ground 61 and the cliff 62. The boundary 63 extends in the traveling direction of the working machine 3. The boundary 63 exists on both sides of the ground 61. The first specific image 70A is generated so as to extend in the traveling direction of the working machine 3 along the boundary 63. The first specific image 70A is a line-shaped image that overlaps the boundary 63. Two first specific images 70A are generated so as to correspond to each of the boundaries 63 existing on both sides of the ground 61. The second specific image 70B is generated so as to surround the obstacle 65. The second specific image 70B is a frame-shaped image that surrounds the obstacle 65.

[0045] The display data generation unit 53 generates display data including a captured image to be displayed on the display device 19 and the specific image 70 superimposed on the captured image. The display data generation unit 53 generates display data when the working machine 3 is traveling on the ground 61. The display data generation unit 53 causes the generated display data to be displayed on the display device 19.

[0046] The captured image is captured by the camera 30. The captured image includes a topographic image showing the topography 60 of the work site 1 in the traveling direction of the work machine 3. The display data generation unit 53 generates display data such that the specific image 70 overlaps the topographic image.

[0047] In the embodiment, the specific image generation unit 52 generates a route image indicating the traveling direction of the work machine 3. The specific image generation unit 52 generates a mesh image showing the three-dimensional shape of the ground 61 on which the work machine 3 travels. The display data generation unit 53 generates display data such that the route image overlaps the topographic image. The display data generation unit 53 generates display data such that the mesh image overlaps the topographic image.

[0048] FIG. 7 is a diagram for explaining a method of generating the route image 80 according to the embodiment. The specific image generation unit 52 sets a specified plane 80V passing through the end 32P of the excavation blade 32. The excavation blade 32 can contact the ground 61 on which the work machine 3 travels. The end 32P includes the left end and the right end of the excavation blade 32. The specified plane 80V is a virtual plane passing through the end 32P and intersecting the surface of the topography 60. The specified plane 80V is parallel to the Xl-Zl plane including the Xl axis and the Zl axis of the local coordinate system (Xl, Yl, Zl) defined on the vehicle body 23 of the work machine 3.

[0049] The local coordinate system (Xl, Yl, Zl) refers to a three-dimensional coordinate system based on the origin defined on the vehicle body 23 of the work machine 3. In the local coordinate system, the front-rear direction, the left-right direction, and the up-down direction are defined. The Xl-axis direction is the front-rear direction. The +Xl direction is the front, and the -Xl direction is the rear. The Yl-axis direction is the left-right direction. The +Yl direction is the left, and the -Yl direction is the right. The rotation axis of the drive wheel 31D of the traveling device 24 extends in the Yl-axis direction. The Yl-axis direction is synonymous with the vehicle width direction of the work machine 3. The Zl-axis direction is the up-down direction. The +Zl direction is the upward, and the -Zl direction is the downward. The ground contact surface of the crawler 31 is orthogonal to the Zl axis.

[0050] The path image 80 shows the intersection where a defined surface 80V passing through end 32P intersects with at least a portion of the surface of the terrain 60 in the direction of travel of the work machine 3. The defined surface 80V is substantially perpendicular to the surface of the terrain 60. In the embodiment, the defined surface 80V is set to be perpendicular to the axis of rotation of the drive wheel 31D of the traveling device 24. The path image 80 includes an intersection line extending in the direction of travel along the surface of the terrain 60.

[0051] The position of the trajectory image 80 is defined in the global coordinate system (Xg, Yg, Zg). The global coordinate system (Xg, Yg, Zg) is a three-dimensional coordinate system based on the origin defined on Earth. The global coordinate system is defined by GNSS (Global Navigation Satellite System). The trajectory image 80 is a collection of multiple intersection points indicating the position in the Xg axis direction, the Yg axis direction, and the Zg axis direction of the surface of the terrain 60. Multiple intersection points are arranged along the surface of the terrain 60 in the direction of travel of the work machine 3. The terrain height shown by the trajectory image 80 is the position in the Zg axis direction of the intersection points. The trajectory image 80 shows the three-dimensional shape of the terrain 60 that the work machine 3 traveling forward will pass through.

[0052] Figure 8 is a diagram illustrating a method for generating a mesh image 90 according to an embodiment. The specific image generation unit 52 generates a mesh image 90 showing the three-dimensional shape of the ground 61 on which the work machine 3 travels, based on the three-dimensional data of the work site 1 acquired by the detection data acquisition unit 50. The mesh image 90 is generated to follow the surface of the terrain 60. The mesh image 90 has a plurality of points 90g indicating the position of the surface of the terrain 60 in a global coordinate system, a first line 90x extending in the Xg axis direction and connecting the plurality of points 90g, and a second line 90y extending in the Yg axis direction and connecting the plurality of points 90g. The points 90g are provided in a matrix on the surface of the terrain 60. Multiple points 90g are provided in the Xg axis direction and multiple points 90g are provided in the Yg axis direction. Each of the plurality of points 90g indicates the position in the Xg axis direction, the Yg axis direction, and the Zg axis direction of the surface of the terrain 60.

[0053] The first line 90x extends in the Xg direction to connect a plurality of points 90g provided in the Xg direction. Multiple first lines 90x are provided at intervals in the Yg direction. The second line 90y extends in the Yg direction to connect a plurality of points 90g provided in the Yg direction. Multiple second lines 90y are provided at intervals in the Xg direction. In this embodiment, multiple first lines 90x are provided at equal intervals in the Yg direction. Multiple second lines 90y are provided at equal intervals in the Xg direction. Points 90g are defined as the intersections of the first line 90x and the second line 90y.

[0054] [Display Method] Figure 9 is a flowchart showing the display method of the display data according to the embodiment. The detection data acquisition unit 50 of the remote controller 20 acquires 3D data of the surroundings of the work machine 3 detected by the 3D sensor 27 of the work machine 3, position data of the work machine 3 detected by the position sensor 28 of the work machine 3, and posture data of the work machine 3 detected by the posture sensor 29 of the work machine 3. When the work machine 3 is traveling on the ground 61, the detection data acquisition unit 50 acquires 3D data of the surroundings of the work machine 3, position data of the work machine 3, and posture data of the work machine 3 that are detected while the work machine 3 is traveling (step S1).

[0055] The detection unit 51 detects a specific part present at the work site 1 based on three-dimensional data of the surroundings of the work machine 3. Based on the three-dimensional data of the surroundings of the work machine 3, the position data of the work machine 3, and the orientation data of the work machine 3, the detection unit 51 can detect the position and size of the specific part in the global coordinate system (step S2).

[0056] If a downward cliff 62 is detected as a specific area, the specific image generation unit 52 generates a first specific image 70A that emphasizes the boundary 63 between the ground 61 and the downward cliff 62. If an obstacle 65 is detected as a specific area, the specific image generation unit 52 generates a second specific image 70B that emphasizes the obstacle 65. Also, as explained with reference to Figure 7, the specific image generation unit 52 generates a path image 80 that shows the direction of travel of the work machine 3. As explained with reference to Figure 8, the specific image generation unit 52 generates a mesh image 90 that shows the three-dimensional shape of the ground 61 on which the work machine 3 is traveling (step S3).

[0057] The detection data acquisition unit 50 acquires the image of the work site 1 captured by the camera 30 of the work machine 3 (step S4).

[0058] The display data generation unit 53 generates display data by superimposing at least one of the specific images 70 (70A, 70B), the path image 80, and the mesh image 90 generated in step S3 with the captured image acquired in step S4. The display data generation unit 53 may also generate display data by superimposing some images selected from the specific images 70 (70A, 70B), the path image 80, and the mesh image 90 with the captured image acquired in step S4. In this embodiment, the display data generation unit 53 generates display data by superimposing all of the specific images 70 (70A, 70B), the path image 80, and the mesh image 90 with the captured image acquired in step S4. The display data generation unit 53 causes the generated display data to be displayed on the display device 19 (step S5).

[0059] Figure 10 shows an example of a display device 19 according to the embodiment. The display data generation unit 53 causes the display device 19 to display an image 100 of the work site 1. The image 100 is captured by the camera 30. The image 100 includes a terrain image showing the terrain 60 of the work site 1 in the direction of travel of the work machine 3. The display data generation unit 53 causes the display device 19 to display display data in which a specific image 70 and the image 100 are superimposed.

[0060] The first specific image 70A highlights the boundary 63 of the captured image 100. The first specific image 70A is displayed superimposed on the boundary 63 of the captured image 100. The boundary 63 extends in the direction of travel of the work machine 3. The boundary 63 exists on both sides of the ground 61. The first specific image 70A is displayed extending along the boundary 63 in the direction of travel of the work machine 3. The first specific image 70A is a line-shaped image superimposed on the boundary 63. Two first specific images 70A are displayed, corresponding to each of the boundaries 63 that exist on both sides of the ground 61. As shown in Figure 10, the presence of the descending cliff 62 may be unclear in the captured image 100. Even if the presence of the descending cliff 62 is unclear in the captured image 100, the operator can recognize the presence of the descending cliff 62 by the first specific image 70A. The operator can operate the work machine 3 so that it does not extend beyond the ground 61.

[0061] The second specific image 70B highlights the obstacle 65 in the captured image 100. The second specific image 70B is displayed so as to surround the obstacle 65 in the captured image 100. The second specific image 70B is a frame-shaped image surrounding the obstacle 65. For example, if the color of the ground 61 and the color of the obstacle 65 are similar, the presence of the obstacle 65 may become unclear in the captured image 100. Even if the presence of the obstacle 65 is unclear in the captured image 100, the operator can recognize the presence of the obstacle 65 by the second specific image 70B. The operator can operate the work machine 3 so that it does not come into contact with the obstacle 65.

[0062] The path image 80 is displayed superimposed on the ground 61 of the captured image 100. The path image 80 is a line-shaped image extending from the end 32P of the excavation blade 32 in the direction of travel of the work machine 3. Two path images 80 are displayed, extending from the left end 32P and the right end 32P of the excavation blade 32 in the direction of travel of the work machine 3. The path image 80 is displayed in a way that it bends to conform to the shape of the ground 61.

[0063] The mesh image 90 is displayed superimposed on the ground 61 of the captured image 100. The first line 90x and the second line 90y of the mesh image 90 are displayed in a way that they are curved to conform to the shape of the ground 61.

[0064] [Effects] As described above, the embodiment includes a detection data acquisition unit 50 that acquires three-dimensional data showing the three-dimensional shape of the work site 1 detected by a three-dimensional sensor 27 of a work machine 3 operating at the work site 1 and an image 100 of the work site 1 captured by a camera 30 of the work machine 3; a detection unit 51 that detects a specific part present in the work site 1 based on the three-dimensional data; a specific image generation unit 52 that generates a specific image 70 showing the specific part; and a display data generation unit 53 that generates display data including an image 100 for display on a display device 19 and a specific image 70 superimposed on the image 100.

[0065] If only the captured image 100 is displayed on the display device 19 in the remote control room 17, it may be difficult for the operator to properly grasp the situation at the work site 1. In this embodiment, a specific image 70 indicating a specific part is superimposed on the captured image 100 of the work site 1, so that the operator in the remote control room 17 can properly grasp the situation at the work site 1. As a result, a decrease in the work efficiency of the work machine 3 is suppressed. The specific part includes a part that requires attention from the operator. Since the specific part is highlighted by the specific image 70, it is easy for the operator to recognize the specific part. This suppresses the occurrence of unforeseen incidents.

[0066] The display data generation unit 53 generates display data at least while the work machine 3 is moving on the ground 61 of the work site 1. The display device 19 displays the display data at least while the work machine 3 is moving on the ground 61 of the work site 1. When the work machine 3 is moving, there is a higher possibility of unforeseen incidents occurring than when the work machine 3 is stationary. At least when the work machine 3 is moving, specific parts are highlighted by specific images 70, so the occurrence of unforeseen incidents is effectively suppressed.

[0067] If a sloping cliff 62 exists near the ground 61 on which the work machine 3 is traveling, a first specific image 70A is displayed that emphasizes the boundary 63 between the ground 61 and the sloping cliff 62. Since the sloping cliff 62 is emphasized by the first specific image 70A, the occurrence of unforeseen incidents is suppressed. The operator can operate the work machine 3 so that it does not extend beyond the ground 61.

[0068] If an obstacle 65 exists in the path of the work machine 3, a second specific image 70B that highlights the obstacle 65 is displayed. Since the obstacle 65 is highlighted by the second specific image 70B, the occurrence of unforeseen incidents is suppressed. The operator can operate the work machine 3 so that it does not come into contact with the obstacle 65.

[0069] The specific image 70 is displayed superimposed on the captured image 100 showing the terrain 60 of the work site 1. The operator can accurately grasp the conditions of the work site 1 based on the captured image 100 of the actual terrain 60 and the specific image 70.

[0070] A path image 80 showing the direction of travel of the work machine 3 is displayed superimposed on an image 100 showing the terrain 60 of the work site 1. Based on the image 100 of the actual terrain 60 and the path image 80, the operator can accurately identify objects in the direction of travel of the work machine 3.

[0071] The path image 80 is a line-shaped image extending from the end 32P of the drilling blade 32 in the direction of travel of the work machine 3. This allows the operator to operate the work machine 3 while checking the width of the drilling blade 32.

[0072] A mesh image 90 showing the three-dimensional shape of the ground 61 on which the work machine 3 travels is displayed superimposed on an image 100 showing the terrain 60 of the work site 1. The operator can accurately understand the terrain 60 of the work site 1 based on the image 100 of the actual terrain 60 and the mesh image 90.

[0073] [Other Embodiments] In the embodiments described above, at least a portion of the functional parts of the remote controller 20 may be provided in the controller 12. For example, the detection unit 51, the specific image generation unit 52, and the display data generation unit 53 may be provided in the controller 12. The specific image 70 and display data generated in the controller 12 may be transmitted to the remote controller 20 and then displayed on the display device 19.

[0074] In the embodiments described above, each of the multiple functional units of the remote controller 20 may be configured by a separate computer (hardware). Each of the multiple functional units of the controller 12 may be configured by a separate computer (hardware). The multiple functional units of the remote controller 20 and the multiple functional units of the controller 12 may be configured by a single computer (hardware).

[0075] In the above-described embodiment, the display device 19 is located outside the work machine 3. The display device 19 may also be located in the operator's cab of the work machine 3. Furthermore, the work machine 3 does not have to be remotely controlled. The work machine 3 may be operated by an operator sitting in the operator's cab of the work machine 3.

[0076] 1...Work site, 2...Remote control system, 3...Work machine (bulldozer), 4...Display system, 10...Communication system, 10A...Wireless communication device, 12...Controller, 17...Remote control room, 18...Remote control device, 19...Display device, 20...Remote controller, 21...Driver's seat, 23...Vehicle body, 24...Running gear, 25...Excavating work machine, 26...Ripper work machine, 27...3D sensor, 27F...3D sensor, 27B...3D sensor, 28...Position sensor, 29...Attitude sensor, 30...Camera, 31...Track, 31D...Drive wheel, 32...Excavating blade, 32A...Cutting edge, 32P...End, 33...Lift frame, 34...Tilt cylinder, 35...Lift cylinder, 36...Shank, 36A...Ripper point, 37...Ripper arm, 38...Tilt cylinder Linda, 39...lift cylinder, 40...beam, 41...detection data transmission unit, 42...operation signal acquisition unit, 43...control unit, 44...processor, 45...main memory, 46...storage, 47...input / output interface, 48...communication interface, 49...computer program, 50...detection data acquisition unit, 51...detection unit, 52...specific image generation unit, 53...display data generation unit, 60...terrain, 61...ground, 62...descending cliff, 63...boundary, 65...obstacle, 70...specific image, 70A...first specific image, 70B...second specific image, 80...path image, 80V...standard surface, 90...mesh image, 90g...point, 90x...first line, 90y...second line, 100...imaging image, DR...detection range, DRF...detection range, DRB...detection range.

Claims

1. A display system comprising a processor, the processor acquires three-dimensional data showing the three-dimensional shape of the work site detected by a three-dimensional sensor of a work machine operating at the work site, acquires an image of the work site captured by a camera of the work machine, detects a specific part present at the work site based on the three-dimensional data, generates a specific image showing the specific part, and generates display data including the captured image and the specific image superimposed on the captured image for display on a display device.

2. The display system according to claim 1, wherein the work machine travels on the ground of the work site, and the processor acquires the three-dimensional data and the captured image at least while the work machine is traveling on the ground of the work site, and generates the display data while the work machine is traveling.

3. The display system according to claim 1, wherein the work machine travels on the ground of the work site, the specific part includes a cliff present at the work site, and the specific image includes an enhanced image that highlights the boundary between the ground and the cliff.

4. The display system according to claim 1, wherein the work machine travels on the ground at the work site, the specific part includes obstacles present at the work site, and the specific image includes an enhanced image that highlights the obstacles.

5. The display system according to claim 1, wherein the work machine travels on the ground of the work site, the captured image includes a terrain image showing the terrain of the work site in the direction of travel of the work machine, and the processor generates the display data such that the specific image is superimposed on the terrain image.

6. The display system according to claim 5, wherein the processor generates a path image indicating the direction of travel of the work machine, and generates the display data such that the path image is superimposed on the terrain image.

7. The display system according to claim 6, wherein the work machine has an excavation work machine including an excavation blade that can contact the ground, and the path image is a line-shaped image extending from the end of the excavation blade in the direction of travel.

8. The display system according to claim 6, wherein the processor generates a mesh image showing the three-dimensional shape of the ground on which the work machine travels, and generates the display data such that the mesh image is superimposed on the terrain image.

9. The display system according to claim 1, wherein the display device is located outside the work machine.

10. A display method comprising: acquiring three-dimensional data showing the three-dimensional shape of a work site detected by a three-dimensional sensor of a work machine operating at a work site; acquiring an image of the work site captured by a camera of the work machine; detecting a specific part present in the work site based on the three-dimensional data; generating a specific image showing the specific part; and generating display data including the captured image and the specific image superimposed on the captured image for display on a display device.

11. The display method according to claim 10, wherein the work machine travels on the ground of the work site, and includes at least acquiring the three-dimensional data and the captured image while the work machine is traveling on the ground of the work site, and generating the display data while the work machine is traveling.

12. The display method according to claim 10, wherein the work machine travels on the ground of the work site, the specific part includes a cliff present at the work site, and the specific image includes an enhanced image that highlights the boundary between the ground and the cliff.

13. The display method according to claim 10, wherein the work machine travels on the ground at the work site, the specific part includes obstacles present at the work site, and the specific image includes an enhanced image that highlights the obstacles.

14. The display method according to claim 10, wherein the work machine travels on the ground of the work site, the captured image includes a terrain image showing the terrain of the work site in the direction of travel of the work machine, and the display data is generated such that the specific image is superimposed on the terrain image.

15. The display method according to claim 14, comprising: generating a path image indicating the direction of travel of the work machine; and generating the display data such that the path image is superimposed on the terrain image.

16. The display method according to claim 15, wherein the work machine has an excavation work machine including an excavation blade that can contact the ground, and the path image is a line-shaped image extending from the end of the excavation blade in the direction of travel.

17. The display method according to claim 15, comprising generating a mesh image showing the three-dimensional shape of the ground on which the work machine travels, and generating the display data such that the mesh image is superimposed on the terrain image.

18. The display method according to claim 10, wherein the display device is located outside the work machine.

Images