Display control system, remote operation system, and display control method

Abstract

This display control system is for a display device and: acquires measurement information obtained through measurement by a work machine; generates and displays a composite image on the display device, the composite image including a reference image representing a reference in the height direction and a ground contact surface height image representing the height of a ground contact surface of the work machine; defines a predetermined height around the work machine which is a height different from the ground contact surface of the work machine as a reference for the reference image; and generates the composite image on the basis of the measurement information such that the ground contact surface height image is in a position corresponding to the height of the reference image.

Description

Indicating a control system, a remote operation system, and a display control method

[0001] The present disclosure relates to a display control system, a remote operation system, and a display control method. This application claims priority to Japanese Patent Application No. 2024-189791 filed in Japan on October 29, 2024, the content of which is incorporated herein by reference.

[0002] Patent Document 1 describes a display control system that controls a display device in a remote cab for remotely operating a work vehicle (hereinafter referred to as a work machine). In the display control system described in Patent Document 1, a supplementary image, which is an image for supplementing the lack of sensation obtained when riding on the work machine, is displayed on the display device. Here, the supplementary image is, for example, a posture image including an image depicting the side surface of the work machine representing the current posture of the work machine.

[0003] International Publication No. 2020 / 090985

[0004] When the work machine performs a loading operation onto the transport vehicle using the remote operation system, it may be difficult for the operator to recognize the height between the work machine and the transport vehicle and the inclination of the work machine on the camera image. That is, the lack of information such as the posture of the work machine becomes a problem. Also, in a manned work machine, the same problem exists for an operator who is not accustomed to the operation.

[0005] An object of the present disclosure is to provide a display control system, a remote operation system, and a display control method that can appropriately supplement information in order to solve the above problems.

[0006] The display control system of the present disclosure is a display control system for a display device, which acquires measurement information measured by a work machine, generates a composite image including a reference image representing a reference in the height direction and a ground contact height image representing the height of the ground contact surface of the work machine, and displays the composite image on the display device. Taking a predetermined height around the work machine, which is different from the ground contact surface of the work machine, as a reference for the reference image, the composite image is generated based on the measurement information such that the ground contact height image is positioned at a position corresponding to the height of the reference image.

[0007] The remote control system of this disclosure comprises the display control system, an operating device for the work machine located at a remote location, and the display device.

[0008] The display control method of the present disclosure is a display control method for a display device, wherein measurement information measured by a work machine is acquired, a composite image is generated including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and the composite image is displayed on the display device, a predetermined height around the work machine which is a different height from the ground surface of the work machine is used as the reference for the reference image, and the composite image is generated such that the ground surface height image is positioned at a position corresponding to the height of the reference image based on the measurement information.

[0009] According to the display control system, remote control system, and display control method of this disclosure, information can be appropriately supplemented by a composite image.

[0010] This is a schematic diagram showing an example configuration of a remote control system according to an embodiment of this disclosure. This is an external view of a work machine according to an embodiment of this disclosure. This is a schematic diagram showing an example of an image captured by an imaging device of a work machine according to an embodiment of this disclosure. This is a schematic block diagram showing an example configuration of a remote control system according to an embodiment of this disclosure. This is a schematic diagram showing an example of an image extracted from an image captured according to an embodiment of this disclosure. This is a schematic diagram showing an example of an image displayed by a display device according to an embodiment of this disclosure. This is a schematic diagram showing an example of a composite image according to an embodiment of this disclosure. This is a schematic diagram showing an example of a composite image according to an embodiment of this disclosure. This is a schematic diagram for explaining a composite image according to an embodiment of this disclosure. This is a schematic diagram showing an example of flowchart showing an example of operation of a remote control system according to an embodiment of this disclosure.

[0011] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding components are given the same reference numerals, and their descriptions will be omitted as appropriate.

[0012] (Remote Control System) Figure 1 is a schematic diagram showing an example configuration of the remote control system 1 according to the embodiment of the present disclosure. Figure 2 is an external view of the work machine 100 according to the embodiment of the present disclosure. Figure 4 is a schematic block diagram showing an example configuration of the remote control system 1 according to the embodiment of the present disclosure. The remote control system 1 is a remote control system for remotely controlling a work machine 100 equipped with a work machine 130 having a work tool, and comprises a work machine 100 that operates by remote control, a transport vehicle 200 such as a dump truck which is the object to be loaded with excavated material for loading by a bucket 133 which is a work tool, and a remote control room 500 for performing remote control. The work machine 100 operates at a work site (for example, a mine, a quarry, etc.). The remote control room 500 is located at a point away from the work machine 100 (a remote location from the work machine 100; for example, an urban area, within the work site, etc.). The work machine 100 and the remote control room 500 are connected via communication means such as a mobile communication network or the internet. The remote control system 1 is a system for operating the work machine 100 using a remote control room 500. The transport vehicle 200 and the remote control room 500 are connected via communication means such as a mobile communication network or the internet, or further connected via an unmanned dump truck operation control system, etc., which will be described later.

[0013] The work machine 100 operates according to operation signals received from the remote control room 500. In other words, there is no operator on board the work machine 100. The remote control room 500 receives the operation of the work machine 100 based on the operator's input and transmits operation signals to the work machine 100. The operation signals will be described later.

[0014] (Remote Control Room) The remote control room 500 includes a driver's seat 510, a first display device 520, a second display device 530, an operating device 540, a control device 550, and a communication device 560. The first display device 520 is positioned in front of the driver's seat 510. The first display device 520 is positioned directly in front of the operator when the operator is seated in the driver's seat 510. As shown in Figure 1, the first display device 520 consists of a central display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525 arranged in a row. The left display 522 is located to the left of the central display 521. The right display 523 is located to the right of the central display 521. The upper display 524 is located above the central display 521. The lower display 525 is located below the central display 521. The first display device 520, for example, divides a portion or all of an image captured by the work machine 100 and displays it on the five displays.

[0015] In other embodiments, the number of displays constituting the first display device 520 is not limited to this. For example, the first display device 520 may be composed of a single display. Also, the first display device 520 may project captured images onto a curved or spherical surface using a projector or the like. Furthermore, the first display device 520 is just one example of the configuration of the "display device" according to this disclosure.

[0016] The second display device 530 is positioned diagonally in front of the driver's seat 510. The second display device 530 displays vehicle information transmitted from the work machine 100 (fuel level, engine water temperature, etc.) and notifications of abnormalities in the work machine 100. In other embodiments, the position of the second display device 530 does not have to be diagonally in front of the driver's seat 510, as long as it is in a position that can be seen by the operator. Furthermore, the remote control room 500 in other embodiments does not have to be equipped with the second display device 530, and preferably the vehicle information displayed on the second display device 530 and notifications of abnormalities are displayed on the first display device 520.

[0017] The operating device 540 is located near the driver's seat 510. The operating device 540 is located within the operator's reach when the operator is seated in the driver's seat 510. The operating device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and electric pedal, the operating device 540 outputs work equipment operation signals, which are operation signals for the boom 131, arm 132, and bucket 133; a slewing operation signal for the slewing body 120; and a travel operation signal for the travel body 110. In this embodiment, the work equipment operation signals, slewing operation signals, and travel operation signals are collectively referred to as operation signals.

[0018] The control device 550 displays the captured images and vehicle body information received from the work machine 100 on the first display device 520. In other words, the control device 550 is an example of a display control system. The control device 550 also transmits the operation signals input to the operation device 540 to the work machine 100. The display control system may be a remote control room 500 including the control device 550, or it may further include a work machine control device 126 and a front camera 122 of the work machine 100. The display control system includes an acquisition unit 552 and a display unit 553. The acquisition unit 552 acquires measurement information such as height information measured by the work machine 100. The display unit 553 generates a composite image P2 including a reference image P25 representing the height reference and a ground surface height image P26 representing the height of the ground surface of the work machine 100, and displays it on the first display device 520. Furthermore, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground contact surface of the work machine 100, as the reference for the reference image P25, and generates a composite image P2 based on the measurement information so that the ground contact surface height image P26 is positioned at a position corresponding to the height of the reference image P25.

[0019] The communication device 560 transmits and receives video signals representing captured images, operation signals, vehicle information, and the like to and from the communication device 143 of the work machine 100.

[0020] (Work Machine) Figure 2 is an external view of a work machine 100 according to an embodiment of the present disclosure. The work machine 100 according to an embodiment of the present disclosure is a backhoe shovel, which is a type of hydraulic excavator. In addition, the work machine 100 according to other embodiments may be other hydraulic excavators such as face shovels or rope shovels, or other work machines such as wheel loaders or bulldozers. The work machine 100 comprises a hydraulically driven work machine 130, a slewing body 120 that supports the work machine 130, and a traveling body 110 that supports the slewing body 120. The slewing body 120 rotates with a slewing axis CR as the rotational axis.

[0021] The work machine 130 comprises a boom 131, an arm 132, and a bucket 133 which is a work tool. The work machine 130 is driven by the extension and retraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. The boom 131, arm 132, and bucket 133 are equipped with a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139, respectively.

[0022] The base end of the boom 131 is attached to the slewing body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via a pin. The bucket 133 includes a cutting edge 133T for excavating soil and other materials, and a container for collecting the excavated soil and other materials. The base end of the bucket 133 is attached to the tip of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. The base end of the boom cylinder 134 is attached to the slewing body 120. The tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. The base end of the arm cylinder 135 is attached to the boom 131. The tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. The base end of the bucket cylinder 136 is attached to the arm 132. The tip end of the bucket cylinder 136 is attached to the bucket 133 via a linkage mechanism.

[0024] The boom angle sensor 137 is attached, for example, to the boom 131 and detects the inclination angle of the boom 131. The arm angle sensor 138 is attached, for example, to the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 139 is attached, for example, to the bucket 133 and detects the inclination angle of the bucket 133. The boom angle sensor 137, arm angle sensor 138, and bucket angle sensor 139 according to the embodiments of this disclosure detect the inclination angle with respect to the ground plane. Note that angle sensors according to other embodiments are not limited to these and may detect the inclination angle with respect to other reference planes. For example, in other embodiments, the angle sensor may detect the relative rotation angle by potentiometers provided at the base ends of the boom 131, arm 132, and bucket 133, or it may detect the inclination angle by measuring the cylinder lengths of the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 and converting the cylinder lengths into angles.

[0025] The slewing body 120 is equipped with a driver's cab 121. The driver's cab 121 is located to the left of the work equipment 130. The driver's cab 121 is equipped with a forward camera 122. The forward camera 122 is installed at the front and top of the driver's cab 121. The forward camera 122 images the area in front of the driver's cab 121 through the windshield at the front of the driver's cab 121. Here, "forward" refers to the direction in which the work equipment 130 is attached to the slewing body 120, and "rear" refers to the opposite direction of "forward". "Side" refers to the direction (left and right) that intersects the front-rear direction. Examples of the forward camera 122 include imaging devices using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. In other embodiments of the work machine 100, the operator's cab 121 may not be provided. In this case as well, the front camera 122 is installed in a position corresponding to the operator's cab 121 to capture images of the area in front. In other embodiments, the front camera 122 may be composed of two or more cameras. The front camera 122 is an example of an imaging device provided in the work machine 100.

[0026] Figure 3 shows an example of an image captured by an imaging device, which is a front camera 122, of a work machine 100 according to an embodiment of this disclosure. The front camera 122 captures the area in front of the work machine 130 and the operator's cab 121 that is the work object. In other words, as shown in Figure 3, the image P1 captured by the front camera 122 shows the work object in front of the work machine 130 and the operator's cab 121. Also, since the operator's cab 121 is located to the left of the work machine 130, a part of the boom 131 is captured in the right portion of the image P1.

[0027] As shown in Figure 2 or Figure 4, the work machine 100 includes a work machine control device 126, a positioning device 123, a front camera 122, an inclination measuring device 124, a hydraulic system 125, a work machine attitude acquisition device 142, and a communication device 143. The work machine attitude acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the rotating body 120 and the direction in which the rotating body 120 is facing. The positioning device 123 is equipped with two receivers that receive positioning signals from artificial satellites that constitute the GNSS (Global Navigation Satellite System). The two receivers are installed at different locations on the rotating body 120. Based on the positioning signals received by the receivers, the positioning device 123 detects information indicating the latitude, longitude, and altitude, as well as the position of the representative point of the rotating body 120 in the field coordinate system (the origin of the shovel coordinate system). Using the positioning signals received by the two receivers, the positioning device 123 calculates the direction in which the rotating body 120 is facing as the relationship between the installation position of one receiver and the installation position of the other receiver. In other embodiments, the positioning device 123 may detect the direction the rotating body 120 is facing based on the measured values ​​of a rotary encoder or an IMU (Internal Measurement Unit).

[0029] The inclinometer 124 measures the acceleration and angular velocity of the slewing body 120 and detects the attitude of the slewing body 120 (for example, the inclinometer angles such as roll angle, pitch angle, and yaw angle) based on the measurement results. The inclinometer 124 is installed, for example, on the underside of the slewing body 120. The inclinometer 124 can be, for example, an inertial measuring unit (IMU). The inclinometer 124 may also be an inclinometer that detects the inclinometer angle independently of acceleration and angular velocity. Note that the work machine 100 according to other embodiments does not need to be equipped with a positioning device 123. Note that Figure 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are all 0 degrees and the directions of the three axes XYZ of the global coordinate system coincide with the directions of the three axes of the field coordinate system (local coordinate system). In this embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are also referred to as the roll angle, pitch angle, and yaw angle of the work machine 100. The X direction is also referred to as the left-right direction, the Y direction as the front-back direction, and the Z direction as the height direction. The inclination angle of the slewing body 120 is also referred to as the inclination angle of the work machine 100.

[0030] The hydraulic system 125 comprises a hydraulic fluid tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by an engine or electric motor (not shown) and supplies hydraulic fluid to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and the position of the spool adjusts the flow rate of hydraulic fluid supplied to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136. The spool is driven based on control commands received from the work machine control device 126. In other words, the amount of hydraulic fluid supplied to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits the captured image P1 taken by the forward camera 122, position information representing the position of the slewing body 120 (including information indicating latitude, longitude, and altitude), direction and inclination angle (attitude), and inclination angle information of the boom 131, as well as the arm 132 and bucket 133, to the remote control room 500 via the communication device 143. The information regarding the inclination angle of the boom 131, as well as the arm 132 and bucket 133, is also referred to as the attitude information of the work machine 130. In this embodiment, the information measured by various sensors on the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives operation signals from the remote control room 500 via the communication device 143. Based on the received operation signals, the work machine control device 126 drives the work machine 130, the slewing body 120, and the traveling body 110.

[0032] (Transport Vehicle) In this embodiment, the transport vehicle 200 is an off-road dump truck equipped with a vessel, and is an example of the transport vehicle 200. In this embodiment, the transport vehicle 200 is operated under the control of an unmanned dump truck operation control system 300. The transport vehicle 200 includes a dump truck control device 201, a positioning device 202, and a communication device 203. The positioning device 202, like the positioning device 123, includes two receivers that receive positioning signals from artificial satellites constituting the GNSS, and acquires the position of the transport vehicle 200 and the direction in which the transport vehicle 200 is facing. The communication device 203 transmits and receives predetermined signals to and from the unmanned dump truck operation control system 300, or to and from the communication device 560 of the remote driver's room 500. The dump truck control device 201 moves the transport vehicle 200 between predetermined points (such as a loading point for excavated material, an unloading point, a waiting area, etc.) under the control of the unmanned dump truck operation control system 300. Furthermore, the dump truck control device 201 repeatedly provides position information and orientation information representing the position of the transport vehicle 200 (including information indicating latitude, longitude, and altitude) at predetermined intervals to the remote control room 500, for example, via the unmanned dump truck operation control device 300.

[0033] (Control device for remote control room) The control device 550 is configured using one or more computers such as a microcontroller, and includes a control unit 551 as a functional block composed of a combination of hardware such as the computer and its peripheral devices, and software such as a program executed by the computer. The control unit 551 also includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0034] The communication unit 554 receives from the work machine 100 via the communication device 560 a video signal representing the captured image P1 taken by the forward camera 122, position information representing the position of the slewing body 120 (information indicating longitude, latitude, and altitude), azimuth and inclination angle, and inclination angle information (also called attitude information) of the boom 131, arm 132, and bucket 133. The communication unit 554 also receives position information representing the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude), and azimuth information from the transport vehicle 200 either directly via the communication device 560 or via the unmanned dump truck operation control device 300, etc. The communication unit 554 also transmits operation signals to the work machine 100 in accordance with the operation of the operation device 540.

[0035] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the captured image P1 taken by the forward camera 122, position information and orientation representing the position of the slewing body 120 (information indicating longitude, latitude, and altitude), attitude information of the work machine 130 representing the inclination angles of the boom 131, arm 132, and bucket 133, and measurement information such as the roll angle and pitch angle of the work machine 100. In addition to acquiring the measurement information, the acquisition unit 552 also acquires position information representing the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude), and measurement information such as orientation information. The acquisition of information indicating the roll angle may be omitted.

[0036] Next, the display unit 553 will be described with reference to Figures 5 to 12. Figure 5 is a schematic diagram showing an example of an image extracted from an captured image P1 according to an embodiment of this disclosure. Figure 6 is a schematic diagram showing an example of an image displayed by the first display device 520 according to an embodiment of this disclosure. Figures 7, 10, and 12 are schematic diagrams showing examples of composite images P2 (P2, P2c, and P2f) according to an embodiment of this disclosure. Figures 8 and 11 are schematic diagrams showing examples of changes in composite images P2 (P2a and P2b, and P2d and P2e) when the tilt angle changes.

[0037] The display unit 553 performs image cropping processing to divide and display the captured image P1 on multiple displays of the first display device 520. The display unit 553 also generates a composite image P2 which includes a reference image P25 representing the height reference of the work machine 100, a ground surface height image P26 representing the height of the ground surface of the work machine 100, a side image P21 representing the side of the work machine 100, and an inclination image P27 representing the inclination angle of the work machine 100, and performs display processing to superimpose the composite image P2 onto one of the cropped images and display it on the first display device 520. In other words, the display unit 553 performs two processes: image cropping processing for dividing and displaying on displays, and display processing to generate a composite image P2 and superimpose the generated composite image P2 onto one of the cropped images and display it on the first display device 520.

[0038] In the image cropping process, the display unit 553 crops the captured image P1 to display the central image P11 on the central display 521, the left image P12 on the left display 522, the right image P13 on the right display 523, the upper image P14 on the upper display 524, and the lower image P15 on the lower display 525, respectively, as shown in Figure 5. Note that if the first display device 520 consists of a single display, cropping of the captured image P1 is not necessary.

[0039] In the display process, the display unit 553 generates composite images P2, P2a to P2f (hereinafter collectively referred to as composite image P2) as illustrated in Figures 7 to 8 and Figures 10 to 12, and displays them on the first display device 520 by superimposing the composite image P2 onto one of the cropped images, for example as shown in Figure 6. In the example shown in Figure 6, the central image P11 is displayed on the central display 521, the left image P12 is displayed on the left display 522, the right image P13 with the composite image P2 superimposed is displayed on the right display 523, the upper image P14 is displayed on the upper display 524, and the lower image P15 is displayed on the lower display 525. In this embodiment, generating an image means, for example, creating an image based on one or more images that simulate the whole or individual parts of a pre-prepared work machine 100, pasting the image onto a predetermined drawing area (for example, a memory area set as a drawing object), drawing the image in raster or vector format, performing modification processing such as coloring, rotating the image, changing the size, orientation or position of the image, and so on.

[0040] The display unit 553 generates a composite image P2 including a reference image P25 representing a height reference and a ground surface height image P26 representing the height of the ground surface 110s of the work machine 100, and displays it on the first display device 520. At this time, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground surface 110s of the work machine 100, as the reference for the reference image P25. In addition, the display unit 553 generates the composite image P2 such that the ground surface height image P26 is positioned at a position corresponding to the height of the reference image P25, based on the measurement information acquired by the acquisition unit 552.

[0041] In this embodiment, as shown in Figure 7, the composite image P2 includes a side view image P21, a grid image P22, scale images P23 and P24, a reference image P25 (see Figure 10, etc.), a ground surface height image P26, an inclination image P27, an inclination angle image P28, and a height image P29. However, in the composite image P2 shown in Figure 7, the reference image P25 and the ground surface height image P26 are displayed overlapping, and for example, only the ground surface height image P26 is visible.

[0042] The side view image P21 is an image of the work machine 100 viewed from the side. The side view image P21 is drawn at a position corresponding to the actual height, based on the reference image P25. In other words, the side view image P21 is displayed at the height indicated by the scale position of the scale image P23, which corresponds to the actual height. The display unit 553 generates a side view image P21 representing the current posture of the work machine 130 based on the tilt angle information of the boom 131, arm 132, and bucket 133 acquired by the acquisition unit 552. The display unit 553 also generates a side view image P21 representing the current posture of the work machine 100 based on the tilt angle of the slewing body 120 acquired by the acquisition unit 552. For example, Figure 7 shows a side view image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is 0°. Furthermore, Figures 8 and 11 show a composite image P2a or P2d including a side image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is 5°, and a composite image P2b or P2e including a side image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is -5°. The display unit 553 generates the composite image P2 such that a portion of the side image P21 is located outside the rectangular frame line P221 included in the grid image P22. In other words, a portion of the side image P21 may be displayed on the outer frame of the grid image P22. In this case (when the tilt angle is 0°), the display unit 553 displays the side image P21 such that the vertical line to the right of the rectangular frame line P221 included in the grid image P22 coincides with the slewing axis CR (Figure 2). Furthermore, the display unit 553 generates a composite image P2 such that the grid image P22, which includes the frame lines P221, is positioned in the forward direction of the side image P21 (in the forward direction of the work machine 100). Alternatively, the side image P21 may be displayed so as to fit within the grid image P22.

[0043] In this embodiment, the grid image P22 includes an image representing a rectangular frame line P221 which is the outer frame of the grid image P22, and a plurality of scale lines P222 arranged in a grid pattern at intervals of 5 m within the frame line P221. The scale image P23 includes numerical values representing the distance in the height direction from the reference height (0 m). In the example shown in FIG. 7, the scale image P23 includes numerical values from -10 m to 25 m at intervals of 5 m. The scale image P24 includes numerical values representing the distance in the front-rear direction with the turning axis CR being 0 m. For example, when the revolving body 120 is inclined as shown in FIG. 8, the scale image P24 includes numerical values representing the distance in the front-rear direction with the intersection point C1 between the turning axis CR and the ground contact surface 110s of the traveling body 110 being 0 m. In the example shown in FIG. 7, the scale image P24 includes numerical values from 0 m to 25 m at intervals of 5 m. The number of scales and the numerical values of the scale images P23 and P24 can be set arbitrarily. In this embodiment, the display unit 553 generates the composite image P2 based on the plurality of scale lines P222.

[0044] The reference image P25 is an image representing the reference in the height direction (0 m). In the example shown in FIG. 10, it is an image representing a line segment thicker than the thickness of the scale line P222 of the grid image P22. Note that the reference image P25 may be, for example, an image with the same thickness as the scale line of the grid image P22 but a different color, or a line type such as a dashed line other than a solid line. The reference of the reference image P25 can be the height of the ground contact surface 110s of the working machine 100 (FIGS. 7 and 8), or a predetermined height around the working machine 100 that is different from the height of the ground contact surface 110s of the working machine 100. The predetermined height around the working machine 100 includes, for example, the height of the traveling path R200 of the transport vehicle 200 (FIGS. 10 to 13).

[0045] The ground contact height image P26 is an image representing the height of the ground contact surface 110s of the work machine 100. In the example shown in FIG. 7, the ground contact height image P26 is shown as a white-painted rectangle with a grid pattern. In this case, the ground contact height image P26 is an image extending in the forward direction of the work machine 100. The ground contact height image P26 is, for example, an image representing a straight line extending in the horizontal direction. Alternatively, when the work machine 100 includes the swivel body 120, the ground contact height image P26 is an image representing a straight line extending horizontally from the intersection point C1 between the swivel center axis (swivel axis CR) of the swivel body 120 and the ground contact surface 110s. The display unit 553 generates the composite image P2 such that the ground contact height image P26 is positioned at a position corresponding to the height of the reference image P25 based on the measurement information acquired by the acquisition unit 552. Note that the reference image P25, the ground contact height image P26, and the inclination image P27 are displayed in different modes. Here, the mode means the display color, thickness, distinction between solid line and broken line, presence or absence of blinking, etc.

[0046] The inclination image P27 is an image representing the inclination angle (pitch angle) of the work machine 100. In the present embodiment, the inclination image P27 is an image representing a broken-line straight line extending from the intersection point C1 in the forward direction of the work machine 100.

[0047] The inclination angle image P28 is an image representing the inclination angle of the inclination image P27 numerically. In the example shown in FIG. 7, the inclination angle image P28 represents the string "Slope: 0°". That is, the inclination angle image P28 shown in FIG. 7 represents that the inclination angle of the swivel body 120 is 0°.

[0048] The height image P29 is an image representing numerically the difference between the height of the reference image P25 and the height of the ground contact height image P26. In the example shown in FIG. 7, the height image P29 represents the string "Height: 0m".

[0049] In this embodiment, as described above, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground contact surface 110s of the work machine 100, as the reference for the reference image P25, and generates a composite image based on the measurement information such that the ground contact surface height image P26 is positioned at a position corresponding to the height of the reference image P25. Here, the measurement information is information measured by predetermined sensors mounted on the work machine 100 or predetermined sensors mounted on the transport vehicle 200, and includes, for example, position information and orientation representing the position of the slewing body 120 (information indicating longitude, latitude, and altitude), attitude information of the work machine 130 representing the inclination angles of the boom 131, arm 132, and bucket 133, attitude information representing the roll angle and pitch angle of the work machine 100, and position information and orientation information representing the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude).

[0050] Furthermore, in this embodiment, when the display unit 553 displays a composite image P2 on the first display device 520, which includes a reference image P25 representing a height reference and a ground surface height image P26 drawn at a position corresponding to the actual height based on the reference image P25, the display unit 553 can generate the composite image P2 by switching the height of the reference image P25 to either the height based on the work machine 100 or a predetermined height around the work machine 100. The actual height is the height measured by various sensors provided by the work machine 100. Here, the predetermined height around the work machine 100 is, for example, the height based on other work machines such as a transport vehicle 200. For example, the predetermined height around the work machine 100 may be the height based on other work machines such as a transport vehicle 200 that are located at a different height from the ground surface of the work machine 100. The display unit 553 may also switch the height of the reference image P25 based, for example, on the positional relationship between the work machine 100 and other work machines. Here, the positional relationship includes relationships relating to absolute positions or relationships relating to relative positions. A relationship relating to absolute positions includes, for example, a relationship in which one or both of the work machine 100 and the other work machine are located in a predetermined location. A relationship relating to relative positions includes, for example, a relationship in which at least one of the horizontal or vertical distances between the work machine 100 and the other work machine is within a predetermined range.

[0051] Furthermore, if the other work machine is a transport vehicle 200, the height based on the other work machine can be made to correspond to the height of the transport vehicle 200's track R200. In addition, the predetermined height around the work machine 100 can be, in addition to the above example, the height of the moving location for excavated material, the target height of the scaffolding when laying scaffolding, etc. Furthermore, the other work machine is not limited to a transport vehicle, but can be any other work machine working in cooperation with the work machine in general. In addition, the display unit 553 may switch the height of the reference image P25 in response to a predetermined input operation by the operator, for example.

[0052] For example, the height of the reference image P25 (image hidden by the ground surface height image P26) included in the composite image P2 shown in Figure 7 corresponds to the height of the ground surface 110s of the traveling body 110 of the work machine 100 shown in Figure 9. Also, for example, the height of the reference image P25 included in the composite image P2c shown in Figure 10 corresponds to the height of the track R200 of the transport vehicle 200 shown in Figure 9. Figure 9 schematically shows the positional relationship between the work machine 100 and the transport vehicle 200. In Figure 9, the work machine 100 is placed on the scaffolding 2. The transport vehicle 200 is located on the track R200 of the transport vehicle 200. The track R200 may be a work site where the transport vehicle 200 stops and the work machine 100 loads cargo onto the transport vehicle 200. In this case, the height of the track R200 of the transport vehicle 200 is lower than the height of the ground surface 110s of the traveling body 110 of the work machine 100. Therefore, the position of the side view image P21 included in the composite image P2c shown in Figure 10 is located above the reference image P25 by the difference in height between the track R200 and the ground surface 110s. In this case, the operator can recognize the height of the track R200 of the transport vehicle 200 by checking the composite image P2c.

[0053] The display unit 553 can switch the height of the reference image P25 based on the positional relationship between the work machine 100 and the transport vehicle 200, for example, as follows. That is, for example, as shown in Figure 9, the display unit 553 uses the height of the track R200 as the reference when the predetermined position P200 of the transport vehicle 200 is located within a predetermined range (within a diameter D200) from the loading position (loading point) of the transport vehicle 200. Alternatively, the display unit 553 can use the predetermined height of the work machine 100 (for example, the height of the ground contact surface 110s of the traveling body 110) as the reference when the predetermined position P200 of the transport vehicle 200 is not located within a predetermined range (within a diameter D200) from the loading position of the transport vehicle 200. The position P200 corresponds to, for example, the installation position of the positioning device 202.

[0054] As described above, Figures 8 and 10-12 show other examples of the composite image P2 shown in Figure 7, which is generated and displayed by the display unit 553. The composite images P2a and P2b shown in Figure 8 differ from the composite image P2 shown in Figure 7 in that the inclination angle of the work machine 100 in the side view image P21 is different. Figures 10-12 differ from Figures 7 and 8 in that the difference between the height of the reference image P25 and the height of the ground surface height image P26 is different. In the cases of Figures 7 and 8, the height difference is 0 m, whereas in the cases of Figures 10-12, the height difference is 4.5 m.

[0055] Furthermore, the shapes of the ground surface height image P26 and the inclination image P27 in Figure 12 differ from those shown in other drawings. In the example shown in Figure 12, when the display unit 553 generates a composite image P2 based on multiple scale lines P222, it generates the composite image P26 such that a part (left end) P261 of the ground surface height image P26 is located outside the frame line P221. Also, the display unit 553 generates the composite image P26 such that a part (left end) P271 of the inclination image P27 is located outside the frame line P221. In other words, the linear images representing the ground surface height image P26 and the inclination image P27 in Figure 12 are longer than those shown in other drawings. By displaying the linear images outside the frame line P221, it becomes easier to grasp the numerical values ​​indicated by the scale image P23.

[0056] (Example of operation of the remote control room) Figure 13 is a flowchart showing the display control method by the control device 550 of the remote control room according to the present disclosure. When remote operation of the work machine 100 is started, the control device 550 executes the following display control at predetermined intervals.

[0057] In the process shown in Figure 13, first, the acquisition unit 552 acquires measurement information, which is the position information of the transport vehicle 200 (step S1). Next, the acquisition unit 552 acquires measurement information such as the captured image P1 taken by the front camera 122 of the work machine 100, position information, pitch angle information, and posture information representing the posture of the work machine 130 (step S2). Next, the display unit 553 extracts the center image P11, left image P12, right image P13, upper image P14, and lower image P15 from the acquired captured image P1 (step S3). Next, the display unit 553 determines whether the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4). If the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4: YES), the display unit 553 determines the height of the transport vehicle 200's track R200 to be the height of the reference image P25 (step S5). On the other hand, if the position of the transport vehicle 200 is not within a predetermined range from the loading position (step S4: NO), the display unit 553 determines a predetermined height of the work machine 100 (for example, the height of the ground surface 110s) to be the height of the reference image P25 (step S6).

[0058] Next, the display unit 553 generates a grid image P22, scale images P23 and P24, and a reference image P25 (step S7). Next, the display unit 553 generates a side view image P21 of the work machine 100 based on the height of the reference image P25, the position information of the work machine 100, the pitch angle, and the posture information representing the posture of the work machine 130 (step S8). Next, the display unit 553 generates a ground surface height image P26 and an inclination image P27 based on the measurement information acquired by the acquisition unit 552 (step S9). Next, the display unit 553 generates an inclination angle image P28 and a height image P29 based on the height of the reference image P25, the position information of the work machine 100, and the pitch angle (step S10). Here, the reference image P25, grid image P22, scale images P23 and P24, side view image P21, ground surface height image P26, inclination image P27, inclination angle image P28, and height image P29 are included in the composite image P2. Next, the display unit 553 superimposes the composite image P2 onto the cropped image and displays it on the first display device 520 (step S11), thus completing the process shown in Figure 13.

[0059] (Function and Effects) As described above, the control device 550 (display control system) according to the embodiment of the present disclosure is a display control system for a first display device 520, and includes an acquisition unit 552 that acquires measurement information measured by the work machine 100, and a display unit 553 that generates a composite image P2 including a reference image P25 representing a height reference and a ground surface height image P26 representing the height of the ground surface 110s of the work machine 100, and displays it on the first display device 520. The display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground surface 110s of the work machine 100, as the reference for the reference image P25, and generates the composite image P2 such that the ground surface height image P26 is positioned at a position corresponding to the height of the reference image P25 based on the measurement information. According to this embodiment and the following embodiments, at least height information can be appropriately supplemented using the composite image P2.

[0060] Furthermore, in this embodiment, the ground contact surface height image P26 is an image that extends in the forward direction of the work machine 100. In other words, the ground contact surface height image P26 is a straight line image that extends in the forward direction of the work machine 100. With this configuration, it is possible to easily grasp the height information in the forward direction of the work machine 100. That is, the ground contact surface height image P26 is a straight line image that extends in the forward direction of the work machine 100 and extends to the frame line P221, so it is possible to easily grasp the numerical value representing the distance from the height reference shown by the scale image P23.

[0061] Furthermore, in this embodiment, the composite image P2 includes a side view of the work machine 100. With this configuration, it is possible to easily grasp the height information of the work machine 100's position in relation to the reference image P25 which represents the height reference.

[0062] Furthermore, in this embodiment, the composite image P2 includes an inclined image P27 that represents the inclination angle of the work machine 100. This configuration makes it easier to grasp the inclination angle of the work machine 100.

[0063] Furthermore, in this embodiment, the ground contact surface height image P26 is an image representing a straight line extending horizontally. With this configuration, the change in the ground contact surface height image P26 can be limited to horizontal movement, thus maintaining a consistent level of ease of recognition of the ground contact surface height image P26.

[0064] Furthermore, in this embodiment, the ground surface height image P26 is an image representing a straight line extending horizontally from the intersection point C1 between the pivot axis (pivot axis CR) of the pivot body 120 and the ground surface 110s. With this configuration, height information can be supplemented based on the pivot axis.

[0065] Furthermore, in this embodiment, the reference image P25, the ground surface height image P26, and the inclination image P27 are each displayed in different ways. This configuration allows for easy identification of each image.

[0066] Furthermore, in this embodiment, the composite image P2 includes an image (grid image P22) representing a rectangular frame line P221 and a plurality of scale lines P222 arranged in a grid pattern within the frame line. The display unit 553 generates the composite image P2 based on the plurality of scale lines P222, and generates the composite image P2 such that a part of the ground surface height image P26 (P261) is located outside the frame line. This configuration makes it easy to quantitatively grasp the difference between the reference height and the ground surface.

[0067] Furthermore, in this embodiment, the display unit 553 generates a composite image P2 such that a portion of the side image P21 is located outside the frame line P221. This configuration allows for efficient use of the area within the frame line P221.

[0068] Furthermore, in this embodiment, the display unit 553 generates a composite image P2 such that the frame line P221 is positioned in the forward direction of the side image P21. With this configuration, height information can be easily obtained for the forward direction of the work machine 100.

[0069] Furthermore, in this embodiment, the predetermined height around the work machine 100, which is used as the reference image P25, can be set to a height based on other work machines. This configuration allows for supplementing height information in relation to other work machines.

[0070] Furthermore, in this embodiment, the work machine 100 is equipped with an imaging device (forward camera 122), and the display unit 553 superimposes a composite image P2 onto the captured image P1 captured by the imaging device and displays it on the first display device 520. With this configuration, the first display device 520 can display the composite image P2 on the captured image P1.

[0071] Furthermore, in this embodiment, the height of the reference image P25 is switched based on the positional relationship between the work machine 100 and other work machines (transport vehicle 200). With this configuration, information can be appropriately supplemented, for example, in work performed in cooperation with other work machines.

[0072] Furthermore, in this embodiment, the height of the reference image P25 is switched according to a predetermined input operation by the operator. With this configuration, information can be appropriately supplemented according to the operator's needs, for example.

[0073] (Other Embodiments) Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes are possible. For example, in the embodiment described above, the control device 550 generates the composite image, but it is not limited to this. For example, in other embodiments, the work machine control device 126 of the work machine 100 or an external server device may generate all or part of the composite image P2, and the control device 550 may receive information representing all or part of the composite image P2.

[0074] Furthermore, while the display control system is implemented in the remote control system 1 in the above-described embodiment, the embodiment is not limited to this. For example, in another embodiment, the control unit 551 may be applied to a radio control system that operates the work machine 100 by wireless communication from a position outside the work machine 100 where the work machine 100 can be seen. When the display control system is applied to the radio control system, the control device 550 may also include a display device. For example, in another embodiment, an operator sitting in the driver's cab 121 of the work machine 100 may view the composite image P2 on a display device provided by the control device 550.

[0075] Furthermore, instead of using the position information of the transport vehicle 200's track R200 as a reference for the position information acquired by the transport vehicle 200 using the positioning device 202, for example, the work machine 100 may be equipped with a LiDAR (Light Detection and Ranging), a stereo camera, etc., to acquire 3D point cloud information of the track R200 (3D position information of multiple measurement points), and the height of the track R200 may be acquired based on the acquired 3D point cloud information with the work machine 100 as the reference. Alternatively, the position information of the transport vehicle 200 may be, for example, the position information acquired by another transport vehicle 200.

[0076] Furthermore, images such as the reference image P25, the ground surface height image P26, and the inclination image P27 do not necessarily have to represent line segments. For example, icons such as arrows and triangles may be used to represent height and inclination angles.

[0077] Furthermore, the predetermined height of the surrounding area may be obtained by recognizing the captured image P1, which is captured using the LiDAR, stereo camera, monocular camera, etc., as described above.

[0078] Furthermore, some or all of the program executed by the computer in the above embodiment can be distributed via a computer-readable recording medium or communication line.

[0079] (Note) The display control system (control device 550), remote control system 1, and display control method relating to this disclosure can be understood, for example, as follows.

[0080] (1) The display control system according to the first embodiment is a display control system for a display device that acquires measurement information measured by a work machine, generates a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displays it on the display device, using a predetermined height around the work machine, which is a different height from the ground surface of the work machine, as the reference for the reference image, and generates the composite image based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image. According to this embodiment and each of the following embodiments, information can be appropriately supplemented.

[0081] (2) The display control system according to the second embodiment is the display control system of (1), wherein the ground surface height image is an image that extends in the forward direction of the work machine.

[0082] (3) The display control system according to the third embodiment is the display control system of (1) or (2), wherein the composite image includes a side view of the work machine.

[0083] (4) The display control system according to the fourth embodiment is the display control system according to (1) to (3), wherein the composite image includes an inclined image representing the inclination angle of the work machine.

[0084] (5) The display control system according to the fifth embodiment is the display control system of (1) to (4), wherein the ground surface height image is an image representing a straight line extending in the horizontal direction.

[0085] (6) The display control system according to the sixth embodiment is the display control system according to (1) to (5), wherein the work machine includes a slewing body, and the ground surface height image is an image representing a straight line extending horizontally from the intersection of the pivot axis of the slewing body and the ground surface.

[0086] (7) The display control system according to the seventh embodiment is the display control system of (1) to (6), wherein the reference image, the ground surface height image, and the inclination image are each displayed in different manners.

[0087] (8) The eighth aspect of the display control system is the display control system of (1) to (7), wherein the composite image includes an image representing a rectangular frame and a plurality of grid lines arranged in a grid within the frame, the composite image is generated based on the plurality of grid lines, and the composite image is generated such that a part of the ground surface height image is located outside the frame.

[0088] (9) The display control system according to the ninth aspect is the display control system according to (1) to (8), wherein the composite image is generated such that a part of the side image is located outside the frame line.

[0089] (10) The display control system according to the tenth embodiment is the display control system of (1) to (9), wherein the display unit generates the composite image such that the frame line is located in the forward direction of the side image.

[0090] (11) The display control system according to the eleventh embodiment is the display control system according to (1) to (10), wherein the predetermined height of the periphery is a height based on other work machines.

[0091] (12) A display control system according to the twelfth embodiment is the display control system according to (1) to (11), wherein the work machine is equipped with an imaging device, and the composite image is superimposed on the image captured by the imaging device and displayed on the display device.

[0092] (13) The display control system according to the 13th embodiment is the display control system according to (1) to (12), wherein the height of the reference image is switched based on the positional relationship between the work machine and the other work machine.

[0093] (14) The display control system according to the 14th embodiment is the display control system according to (1) to (13), wherein the height of the reference image is switched in accordance with a predetermined input operation of the operator.

[0094] (15) The remote control system according to the 15th embodiment comprises the display control system of (1) to (14), the operating device of the work machine installed at a remote location, and the display device.

[0095] (16) A display control method according to the 16th embodiment is a display control method for a display device, wherein measurement information measured by a work machine is acquired, a composite image is generated including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and the composite image is displayed on the display device, a predetermined height around the work machine which is a different height from the ground surface of the work machine is used as the reference for the reference image, and the composite image is generated based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image.

[0096] According to the display control system, remote control system, and display control method of this disclosure, information can be appropriately supplemented by a composite image.

[0097] 1...Remote control system 100...Work machine 130...Work machine 120...Slewing body 110...Traveling body 131...Boom 132...Arm 133...Bucket 134...Boom cylinder 135...Arm cylinder 136...Bucket cylinder 137...Boom angle sensor 138...Arm angle sensor 139...Bucket angle sensor 121...Operator's cab 122...Forward camera 123...Positioning device 124...Incline measuring device 125...Hydraulic system 126...Work machine control device 200...Transport vehicle 202...Positioning device 500...Remote control cab 510...Driver's seat 520...First display device 530...Second display device 540...Operating device 550...Control device 551...Control unit 552...Acquisition unit 553...Display unit 554...Communication unit P2, P2a~P2f...Composite image P21...Side view image P22...Grid image P25...Reference image P26...Ground surface height image P27...Inclined image

Claims

1. A display control system for a display device, comprising: acquiring measurement information measured by a work machine; generating a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displaying it on the display device; using a predetermined height around the work machine, which is a different height from the ground surface of the work machine, as the reference for the reference image; and generating the composite image based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image.

2. The display control system according to claim 1, wherein the ground surface height image is an image that extends in the forward direction of the work machine.

3. The display control system according to claim 2, wherein the composite image includes a side view of the work machine.

4. The display control system according to claim 3, wherein the composite image includes an inclined image representing the inclination angle of the work machine.

5. The display control system according to claim 1, wherein the ground surface height image is an image representing a straight line extending in the horizontal direction.

6. The display control system according to claim 5, wherein the work machine includes a slewing body, and the ground surface height image is an image representing a straight line extending horizontally from the intersection of the pivot axis of the slewing body and the ground surface.

7. The display control system according to claim 4, wherein the reference image, the ground surface height image, and the inclination image are each displayed in different manner.

8. The display control system according to claim 7, wherein the composite image includes an image representing a rectangular frame and a plurality of grid lines arranged in a grid within the frame, the composite image is generated based on the plurality of grid lines, and the composite image is generated such that a portion of the ground surface height image is located outside the frame.

9. The display control system according to claim 8, which generates the composite image such that a portion of the side view image is located outside the frame line.

10. The display control system according to claim 9, which generates the composite image such that the frame line is positioned in the forward direction of the side image.

11. The display control system according to claim 10, wherein the predetermined height of the surrounding area is a height based on other work machines.

12. The display control system according to claim 11, wherein the work machine is equipped with an imaging device, and the composite image is superimposed on the image captured by the imaging device and displayed on the display device.

13. The display control system according to claim 12, which switches the height of the reference image based on the positional relationship between the work machine and the other work machine.

14. The display control system according to claim 13, which switches the height of the reference image in response to a predetermined input operation by an operator.

15. A remote control system comprising the display control system according to any one of claims 1 to 14, an operating device for the work machine located at a remote location, and the display device.

16. A display control method for a display device, comprising: acquiring measurement information measured by a work machine; generating a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displaying it on the display device; a predetermined height around the work machine, which is a different height from the ground surface of the work machine, being used as the reference for the reference image; and generating the composite image such that the ground surface height image is positioned at a position corresponding to the height of the reference image based on the measurement information.

Images