Construction machine display system

Abstract

The purpose is to provide a construction machine display system capable of preventing degradation of work efficiency, by displaying an image that enables an operator to easily grasp perspective. This construction machine display system is provided with a control device for exercising display control for displaying a three-dimensional image of an object by using a captured image of the object acquired by an imaging device attached to the construction machine, and a display device for displaying the three-dimensional image under the display control of the control device. The control device changes a focal position of the three-dimensional image on the basis of the posture or work details of a work device. The display device displays the three-dimensional image of which the focal position has been changed.

Description

Display system for construction machinery

[0001] The present invention relates to a display system for construction machinery.

[0002] At disaster sites and other locations where it is difficult for humans to enter, construction machinery that does not require an operator on board and is remotely operated while the operator remotely views two-dimensional images captured at the site is used. For example, Patent Document 1 proposes an image display system for a working machine, which generates an image of a portion corresponding to a working tool on a work target facing the working tool, combines the image with the image of the work target captured by an imaging device, and causes the combined image to be displayed on a display device.

[0003] Japanese Patent No. 7131779

[0004] For example, a hydraulic excavator as a construction machine includes a working device that performs operations such as excavating or leveling earth and sand. This working device has a boom, an arm, and a bucket, and can secure a wide working range by changing their postures, allowing operations on objects close to the construction machine as well as objects far from the construction machine. Furthermore, among the work contents of the working device, there are operations that require fine manipulation of the working device such as leveling work, and operations that do not require fine manipulation such as excavation work. In such a situation, even if an operator attempts to operate while viewing a two-dimensional image, it is difficult to grasp perspective in the two-dimensional image, and it is difficult to intuitively grasp the positional relationship between the working device and the target object. As a result, the working efficiency of the construction machinery may decrease.

[0005] The image display system disclosed in Patent Document 1 merely displays an image synthesized by projecting a working tool onto an object in a two-dimensional image lacking in perspective, and cannot display an image that allows the operator to easily grasp perspective. Therefore, with the image display system disclosed in Patent Document 1, it is difficult for an operator to intuitively grasp the positional relationship between the working device and a target object, which may reduce the working efficiency of the construction machinery.

[0006] In view of the foregoing circumstances, an object of the present invention is to provide a display system for construction machinery that can suppress a decrease in working efficiency by displaying an image that allows an operator to easily grasp perspective.

[0007] To solve the aforementioned problems, the display system for construction machinery of the present invention comprises a control device that performs display control to display a three-dimensional image of an object using an image of the object acquired by an imaging device attached to the construction machinery, and a display device that displays the three-dimensional image based on the display control of the control device, wherein the control device changes the focal position of the three-dimensional image based on the posture or work content of the work device, and the display device displays the three-dimensional image with the changed focal position.

[0008] According to the present invention, a display system for construction machinery can be provided that suppresses a decrease in work efficiency by displaying images that make it easier for the operator to grasp depth perception.

[0009] Figure showing a construction machine and remote control device to which the display system of this embodiment is applied. Block diagram showing the configuration of the display system of this embodiment. Figure showing an example of the judgment criteria of the judgment unit shown in Figure 2. Figure showing another example of the judgment criteria of the judgment unit shown in Figure 2. Figure showing another example of the judgment criteria of the judgment unit shown in Figure 2. Figure showing another example of the judgment criteria of the judgment unit shown in Figure 2. Figure illustrating the mounting locations of the first stereo camera and the second stereo camera shown in Figure 2. Front view of the first stereo camera shown in Figure 7. Front view of the second stereo camera shown in Figure 7. Figure illustrating another embodiment of the imaging device. Sequence diagram illustrating the operation of the construction machine and remote control device shown in Figure 2.

[0010] Embodiments of the present invention will be described below with reference to the drawings. Components denoted by the same reference numerals in each embodiment are similar in each embodiment unless otherwise specified, and their descriptions will be omitted.

[0011] Figure 1 shows a construction machine 1 and a remote control device 30 to which the display system 100 of this embodiment is applied.

[0012] Construction machine 1 may be a hydraulic excavator or a wheel loader, among other construction machines. Figure 1 shows a hydraulic excavator as an example of construction machine 1, but the type of construction machine 1 is not particularly limited. Furthermore, the construction machine 1 shown in Figure 1 is an unmanned construction machine operated by a remote control device 30 located at a remote location away from the construction site. However, it is not limited to this, and construction machine 1 may be a manned construction machine operated by an operator riding on the construction machine 1.

[0013] The construction machine 1 shown in Figure 1 comprises a self-propelled lower traveling body 3, an upper rotating body 4 rotatably mounted on the lower traveling body 3, and a work device 5 rotatably connected to the front of the upper rotating body 4. In this embodiment, the lower traveling body 3 and the upper rotating body 4 are collectively referred to as "vehicle body 3, 4".

[0014] The upper slewing body 4 is part of the vehicle body 3 and 4 of the construction machine 1 and rotatably supports the work device 5. The upper slewing body 4 is equipped with an engine 6 as the prime mover and a main pump 7, which is a hydraulic pump driven by the engine 6. In the construction machine 1, the lower traveling body 3, the upper slewing body 4, and the work device 5 operate independently using the hydraulic fluid supplied by the main pump 7.

[0015] The work device 5 is a device that performs various operations such as excavating or leveling soil, or loading excavated soil onto other machinery (for example, a dump truck). The work device 5 comprises a boom 10 with one end rotatably connected to the upper slewing body 4, an arm 11 with one end rotatably connected to the other end of the boom 10, and a bucket 12 with one end rotatably connected to the other end of the arm 11.

[0016] The boom 10 rotates vertically around its connection point with the upper slewing body 4 by a boom cylinder 13 driven by hydraulic fluid supplied from the main pump 7. The arm 11 rotates longitudinally around its connection point with the boom 10 by an arm cylinder 14 driven by hydraulic fluid supplied from the main pump 7. The bucket 12 rotates longitudinally around its connection point with the arm 11 by a bucket cylinder 15 driven by hydraulic fluid supplied from the main pump 7.

[0017] Furthermore, a cabin (operator's cab) 8 is provided at the front of the upper rotating body 4. An imaging device 9 is mounted in the cabin 8 so that it can photograph the area in front of the upper rotating body 4. The images acquired by the imaging device 9 are transmitted to a remote control device 30, which will be described later.

[0018] Furthermore, the upper slewing body 4 and the work device 5 are equipped with a posture detection device 16 that detects the posture of the construction machine 1, particularly the posture of the work device 5. For example, the posture detection device 16 is composed of inertial measurement units (IMUs) attached to the boom 10, arm 11, and bucket 12, respectively. The posture detection device 16 detects the posture (e.g., angle to the ground) of the boom 10, arm 11, and bucket 12. The posture information of the construction machine 1 (particularly the work device 5), which is the result of the posture detection device 16, is transmitted to the remote control device 30. The posture information is used to calculate the tip position (toe position) of the bucket 12. Note that the posture detection device 16 may be composed of other sensors if a similar effect can be obtained. The posture detection device 16 may also include a sensor that detects the slewing angle of the upper slewing body 4, and sensors that detect the inclination angles of the vehicle bodies 3 and 4.

[0019] The remote control device 30 is a device for remotely controlling the construction machine 1. The remote control device 30 comprises an operating device 31 consisting of operating levers and operating pedals for the operator to remotely control the construction machine 1, and a display device 32 that displays captured images acquired by the imaging device 9. The operator can remotely control the construction machine 1, which is located far from the remote control device 30, by operating the operating device 31 while viewing the image displayed on the display device 32. The operating device 31 includes an operating amount detection device that detects the amount and direction of operation of the operating levers and operating pedals, and transmits the operation information, including the detected amount and direction of operation, to a control device 34, which will be described later.

[0020] Figure 2 is a block diagram showing the configuration of the display system 100 in this embodiment. Figure 3 is a diagram showing an example of the judgment criteria of the judgment unit 34B shown in Figure 2.

[0021] The display system 100 is a system that displays the object being worked on by the work device 5 of the construction machine 1. For example, when excavating soil or other material with the bucket 12 that constitutes the work device 5, the bucket 12 and the excavated material are located in approximately the same position, so the bucket 12 itself can also be considered the object.

[0022] The display system 100 includes an imaging device 9 that photographs the front of the construction machine 1, an image processing device 17 that selects an image to be displayed on the display device 32 from the images acquired by the imaging device 9, an attitude detection device 16 that detects the attitude of the work device 5, communication devices 18 and 33 that communicate between the construction machine 1 and the remote control device 30, a display device 32 that displays the image selected by the image processing device 17 as a two-dimensional or three-dimensional image, and a control device 34 that controls the display of the image shown on the display device 32.

[0023] In the display system 100, the imaging device 9, posture detection device 16, image processing device 17, and communication device 18 are provided on the construction machine 1. The display device 32, communication device 33, and control device 34 are provided on the remote control device 30.

[0024] The imaging device 9 is configured with stereo cameras so that a three-dimensional image of an object can be displayed on the display device 32. The stereo cameras consist of at least two monocular cameras spaced apart from each other in the horizontal direction (left-right direction of the construction machine 1). Furthermore, the imaging device 9 is configured with a plurality of stereo cameras 21, 22 with different working distances so that three-dimensional images of objects near the vehicle bodies 3, 4 being worked on by the bucket 12 constituting the work device 5 and three-dimensional images of objects far from the vehicle bodies 3, 4 can be displayed appropriately. The working distance is the distance from the stereo cameras 21, 22 to the bucket 12 constituting the work device 5 (for example, to the tip position of the bucket 12), or the distance from the stereo cameras 21, 22 to the object being worked on by the bucket 12. The stereo cameras 21, 22 have different focal positions depending on their different working distances.

[0025] In this embodiment, the focal position is the position where the stereo cameras 21 and 22 focus. In this embodiment, focusing means that, with the object being worked on by the bucket 12 or the bucket 12 itself as the focal position, the surrounding display image does not become excessively blurred, and the operator can perceive whether the focused object or bucket 12 is located near or far from the vehicle body 3 and 4, while also being able to clearly see the focused object or bucket 12.

[0026] The multiple stereo cameras 21 and 22 include a first stereo camera 21 with a predetermined working distance and a second stereo camera 22 with a working distance longer than the predetermined value. That is, the first stereo camera 21 is a stereo camera for acquiring images of objects near the vehicle bodies 3 and 4 so that a three-dimensional image focused on objects near the vehicle bodies 3 and 4 can be displayed. The second stereo camera 22 is a stereo camera for acquiring images of objects far from the vehicle bodies 3 and 4 so that a three-dimensional image focused on objects far from the vehicle bodies 3 and 4 can be displayed. Details of the first stereo camera 21, the second stereo camera 22, and the imaging device 9 will be described later with reference to Figures 7 to 9.

[0027] The image processing device 17 selects an image to be used for display control of the image displayed on the display device 32, based on a command from the control device 34. Specifically, the image processing device 17 operates the first stereo camera 21 or the second stereo camera 22 to acquire an image using the stereo cameras 21 and 22 specified in the command from the control device 34. The image processing device 17 transmits the image acquired by the specified stereo cameras 21 and 22 to the communication device 18, which then transmits it to the remote control device 30.

[0028] Communication devices 18 and 33 are devices for communicating between the construction machine 1 and the remote control device 30 via wireless communication. The communication device 18 of the construction machine 1 transmits the captured image selected by the image processing device 17 and the posture information detected by the posture detection device 16 to the communication device 33 of the remote control device 30. Upon receiving this information, the communication device 33 of the remote control device 30 transmits it to the control device 34. The communication device 33 of the remote control device 30 also receives a command from the control device 34 to select the captured image used for display control of the image displayed on the display device 32, and transmits it to the communication device 18 of the construction machine 1. Upon receiving this command, the communication device 18 of the construction machine 1 transmits it to the image processing device 17.

[0029] In this embodiment, only the part relating to the display control of the image displayed on the display device 32 has been described. However, the communication device 18 and the communication device 33 can send and receive various types of information, including operation information of the operating device 31 operated by the operator.

[0030] The display device 32 is positioned opposite the operator's seat so that the operator can operate the control device 31 while viewing the display device 32. The display device 32 consists of a display capable of displaying two-dimensional and three-dimensional images. The operator can view the three-dimensional images with the naked eye or by wearing special glasses. By displaying three-dimensional images on the display device 32, the operator can perceive depth, which helps to suppress a decrease in work efficiency even when working remotely. The display device 32 displays a three-dimensional or two-dimensional image of an object through display control by the control device 34.

[0031] The control device 34 comprehensively controls each component of the construction machine 1 and the remote control device 30 to control the operation of the construction machine 1. Specifically, the control device 34 generates operation commands corresponding to the operation information of the control device 31 so that the construction machine 1 operates in response to said operation information. The control device 34 transmits the generated operation commands to the communication device 33, which then transmits them to the construction machine 1.

[0032] Furthermore, the control device 34 calculates the tip position of the bucket 12 (hereinafter also referred to as the "bucket position") based on the posture information detected by the posture detection device 16 and transmitted to the remote control device 30, and the lengths of the boom 10, arm 11, and bucket 12, which are stored in advance. Based on the bucket position and the operation information of the control device 31, the control device 34 determines the work being performed by the work device 5 on the construction machine 1.

[0033] Furthermore, the control device 34 uses the image of the object acquired by the imaging device 9 to perform display control to display a three-dimensional image of the object on the display device 32. At this time, the control device 34 changes the focal position of the three-dimensional image based on the posture of the work device 5 or the work content. To realize this function, the control device 34 includes a control unit 34A that performs display control of the three-dimensional image displayed on the display device 32, and a determination unit 34B that determines the focal position of the three-dimensional image displayed on the display device 32.

[0034] For example, the control unit 34A uses the images acquired by the monocular cameras of the first stereo camera 21 or the second stereo camera 22 that constitute the imaging device 9 as the right-eye image and the left-eye image, and controls the display of the three-dimensional image shown on the display device 32 using a display control method such as a passive stereo method or an active stereo method. The control unit 34A may also control the display of the three-dimensional image by other methods if a similar effect can be obtained.

[0035] The determination unit 34B determines the focal position of the three-dimensional image displayed on the display device 32 based on the posture of the work device 5 or the work content. The focal position of the three-dimensional image is, as described above, the position of the object in the three-dimensional image that is focused on so that the outline of the object in the three-dimensional image is clearly displayed and the operator can clearly see the object.

[0036] For example, as shown in Figure 3, if the bucket position is within a first range close to the vehicle body 3 and 4, the determination unit 34B determines that the focal position of the 3D image should be set to the position of an object within the first range close to the vehicle body 3 and 4. On the other hand, if the bucket position is within a second range far from the vehicle body 3 and 4, the determination unit 34B determines that the focal position of the 3D image should be set to the position of an object within the second range far from the vehicle body 3 and 4. In this way, the determination unit 34B can change the focal position of the 3D image according to the bucket position.

[0037] The bucket position is important because the operator operates the construction machine 1 while observing the bucket position. By aligning the focal point of the 3D image with the bucket position, the control device 34 makes it easier for the operator to grasp the sense of depth, thus making it easier to intuitively understand the positional relationship between the bucket 12 and the object, and thus suppressing a decrease in work efficiency.

[0038] The determination unit 34B determines the imaging device 9 from which the captured image used for display control of the 3D image will be acquired, such that the focal position of the 3D image becomes the determined focal position. Specifically, the determination unit 34B selects one stereo camera from among a plurality of stereo cameras 21 and 22 as the imaging device 9 from which the captured image used for display control of the 3D image will be acquired. More specifically, if the bucket position is within a first range close to the vehicle body 3 and 4, the determination unit 34B selects the first stereo camera 21 with a short working distance as the imaging device 9 from which the captured image used for display control of the 3D image will be acquired. On the other hand, if the bucket position is within a second range far from the vehicle body 3 and 4, the determination unit 34B selects the second stereo camera 22 with a long working distance as the imaging device 9 from which the captured image used for display control of the 3D image will be acquired. The determination unit 34B then notifies the control unit 34A of the information of the selected stereo cameras 21 and 22 as the determination result of the determination unit 34B. The control unit 34A sends a command to the communication device 33 to specify the stereo cameras 21 and 22 that will acquire the captured images, so that the captured images are acquired by the stereo cameras 21 and 22 indicated by the determination result of the determination unit 34B, and the communication device 33 transmits this command to the construction machine 1.

[0039] In this way, the control device 34 can change the focal position of the 3D image by selecting one stereo camera from among the multiple stereo cameras 21 and 22 as the imaging device 9 that acquires the captured image used for controlling the display of the 3D image, based on the posture or work content of the work device 5. The display device 32 can then display the 3D image whose focal position has been changed by the control device 34.

[0040] Furthermore, the determination criteria of the determination unit 34B, which determines the focal position of the 3D image, may be defined in three or more stages, rather than just two stages, as shown in Figure 3, where the bucket position is close to / far from the vehicle body 3 and 4. Also, the determination criteria of the determination unit 34B may be the examples shown in Figures 4 to 6, rather than the example shown in Figure 3. The determination criteria of the determination unit 34B may be anything else as long as a similar effect is obtained.

[0041] Figure 4 shows another example of the judgment criteria of the judgment unit 34B shown in Figure 2. Figure 5 shows another example of the judgment criteria of the judgment unit 34B shown in Figure 2. Figure 6 shows another example of the judgment criteria of the judgment unit 34B shown in Figure 2.

[0042] As shown in Figure 4, the determination criteria of the determination unit 34B may not be defined in two stages, such as whether the bucket position is close to or far from the vehicle body 3 and 4, as shown in Figure 3, but rather may be defined such that the further the bucket position is from the vehicle body 3 and 4, the further the focal position of the 3D image becomes from the vehicle body 3 and 4. In other words, the determination unit 34B can change the focal position of the 3D image to a position further from the vehicle body 3 and 4 as the bucket position is further from the vehicle body 3 and 4. This makes it easier for the operator to grasp the perspective of the tip position of the bucket 12, making it easier to intuitively grasp the positional relationship between the bucket 12 and the object, and thus suppressing a decrease in work efficiency.

[0043] As shown in Figure 5, the determination criteria of the determination unit 34B may be defined according to the work content. If the work content is a loading operation performed within a first range close to the vehicle body 3, 4, the determination unit 34B determines that the focal position of the 3D image should be set to the position of the object within the first range close to the vehicle body 3, 4. On the other hand, if the work content is an excavation operation performed within a second range far from the vehicle body 3, 4, the determination unit 34B determines that the focal position of the 3D image should be set to the position of the object within the second range far from the vehicle body 3, 4. In this way, the determination unit 34B can change the focal position of the 3D image according to the work content.

[0044] Note that the work content used for the determination criteria of the determination unit 34B is not limited to being defined in two stages, i.e., excavation work or loading work, and may be defined in three or more stages. Further, the work content used for the determination criteria of the determination unit 34B is not limited to that shown in Fig. 5, and may be any other content as long as the same effect can be obtained.

[0045] When the three-dimensional image displayed on the display device 32 has a single focal position, the workable range of the work device 5 is wide, and thus an operator performing remote operation may not be able to sufficiently grasp the sense of perspective. When the focal position of the three-dimensional image is close to the vehicle bodies 3, 4, it is difficult for the operator to grasp the sense of perspective when performing excavation work in a second range far from the vehicle bodies 3, 4. Conversely, when the focal position of the three-dimensional image is far from the vehicle bodies 3, 4, it is difficult for the operator to grasp the sense of perspective when performing loading work in a first range close to the vehicle bodies 3, 4. Therefore, in such a case, there is a possibility that the work efficiency of the construction machine 1 decreases. The control device 34 can change the focal position of the three-dimensional image according to the work content, thereby making it easy for the operator to grasp the sense of perspective and suppressing a decrease in work efficiency.

[0046] Further, the control device 34 may not only change the focal position of the three-dimensional image according to the work content, but also perform display control to switch between the three-dimensional image and the two-dimensional image according to the work content. The display device 32 may switch and display the three-dimensional image and the two-dimensional image according to the display control of the control device 34.

[0047] In this case, the determination unit 34B determines whether an image displayed on the display device 32 is to be a three-dimensional image or a two-dimensional image in accordance with, for example, determination criteria as shown in Fig. 6. Specifically, as shown in Fig. 6, when the work content requires high precision, the determination unit 34B determines that the image displayed on the display device 32 is to be a three-dimensional image. On the other hand, when the work content does not require high precision, the determination unit 34B determines that the image displayed on the display device 32 is to be a two-dimensional image.

[0048] The work performed by the working device 5 includes work that requires fine precision in the operation of the working device 5 and work that does not require fine precision. Examples of work that requires fine precision include leveling work, demolition work, and attachment replacement (replacement of the bucket 12) work. An example of work that does not require fine precision is excavation work. In the case of work that requires fine precision, a three-dimensional image is displayed on the display device 32 so that an operator can grasp perspective. In the case of work that does not require fine precision, a two-dimensional image is displayed on the display device 32, which reduces the amount of information transmitted from the image processing device 17 to the control device 34 and the display device 32. This is because a three-dimensional image is displayed using captured images from two monocular cameras that constitute the first stereo camera 21 or the second stereo camera 22, whereas a two-dimensional image only needs to be displayed using a captured image from one monocular camera that constitutes the first stereo camera 21 or the second stereo camera 22.

[0049] Figure 7 is a diagram illustrating the attachment locations of the first stereo camera 21 and the second stereo camera 22 shown in Fig. 2.

[0050] The stereo cameras 21 and 22 constituting the imaging device 9 are attached to the front upper part and the front lower part of the upper revolving structure 4. Specifically, the first stereo camera 21 is attached to the front lower part of the cabin 8 mounted on the upper revolving structure 4. The second stereo camera 22 is attached to the front upper part of the cabin 8. That is, the second stereo camera 22 having a longer working distance is attached to a higher position than the first stereo camera 21 having a shorter working distance. This is because the front lower part of the cabin 8 is more suitable for photographing a first range that is closer to the vehicle bodies 3 and 4 than the front upper part of the cabin 8, while the front upper part of the cabin 8 is more suitable for photographing a second range that is farther from the vehicle bodies 3 and 4 than the front lower part of the cabin 8.

[0051] The first stereo camera 21 and the second stereo camera 22 may be configured to change their elevation or depression angles by a mechanical mechanism (not shown). The elevation or depression angles of the first stereo camera 21 and the second stereo camera 22 may be changed to match the bucket position. This allows the operator to constantly view images near the bucket position on the display device 32, making remote operation easier.

[0052] Figure 8A is a view of the first stereo camera 21 shown in Figure 7 from the front. Figure 8B is a view of the second stereo camera 22 shown in Figure 7 from the front.

[0053] As shown in Figure 8A, the first stereo camera 21 is composed of two monocular cameras 21A and 21B arranged horizontally (in the left-right direction of the construction machine 1) with a distance L1 between them. As shown in Figure 8B, the second stereo camera 22 is composed of two monocular cameras 22A and 22B arranged horizontally (in the left-right direction of the construction machine 1) with a distance L2 between them.

[0054] The distance L2 between the monocular cameras 22A and 22B that make up the second stereo camera 22, which has a long working distance, is longer than the distance L1 between the monocular cameras 21A and 21B that make up the first stereo camera 21, which has a short working distance. In stereo cameras 21 and 22, the working distance can be increased by increasing this distance L1 and L2 (baseline length), and the working distance can be decreased by decreasing this distance L1 and L2 (baseline length).

[0055] When the control device 34 sets the focal position of the 3D image to be far from the vehicle body 3 and 4, it selects the second stereo camera 22, which has a long working distance, and has the second stereo camera 22 acquire the captured image. On the other hand, when the control device 34 sets the focal position of the 3D image to be close to the vehicle body 3 and 4, it selects the first stereo camera 21, which has a short working distance, and has the first stereo camera 21 acquire the captured image.

[0056] Figure 9 illustrates another embodiment of the imaging device 9.

[0057] The imaging device 9 may consist of three or more monocular cameras 23 instead of multiple stereo cameras 21, 22. Figure 9 shows three monocular cameras 23A to 23C arranged at intervals from each other in the horizontal direction (left-right direction of the construction machine 1). When the control device 34 sets the focal position of the 3D image to be far from the vehicle body 3, 4, it selects the left monocular camera 23A and the right monocular camera 23B, and causes the two selected monocular cameras 23A and 23B to acquire images. On the other hand, when the control device 34 sets the focal position of the 3D image to be close to the vehicle body 3, 4, it selects the left monocular camera 23A or the right monocular camera 23B and the intermediate monocular camera 23C, and causes the two selected monocular cameras 23C and monocular camera 23A or 23B to acquire images.

[0058] In the example shown in Figure 9, the imaging device 9 was composed of three monocular cameras 23A to 23C, but it may also be composed of four or more monocular cameras 23. In this case, the control device 34 can select two monocular cameras 23 such that the distance between them increases as the focal point of the three-dimensional image moves further away from the vehicle body 3, 4, and then have the two selected monocular cameras 23 acquire the captured image.

[0059] In this way, the control device 34 can change the focal position of the 3D image by selecting two monocular cameras 23 from among three or more monocular cameras 23 as the imaging device 9 that acquires the captured images used for controlling the display of the 3D image, based on the posture or work content of the work device 5. As a result, the display system 100 can reduce the number of monocular cameras compared to when multiple stereo cameras 21 and 22 are used, and can display a 3D image that makes it easy for the operator to grasp the sense of depth with a simple configuration.

[0060] Furthermore, the imaging device 9 may be configured to include two or more monocular cameras 23 and a moving mechanism 24 that moves the monocular cameras 23 so that the distance between them changes in the left-right direction of the construction machine 1. In the example of Figure 9, the intermediate monocular camera 23C is movable in the left-right direction by the moving mechanism 24. In this case, the control device 34 can, for example, select monocular camera 23A and monocular camera 23C, and move monocular camera 23C to the right so that the distance between monocular cameras 23A and 23C increases as the focal position of the 3D image moves further away from the vehicle body 3,4, thereby allowing monocular cameras 23A and 23C to acquire images.

[0061] In this way, the control device 34 can change the focal position of the 3D image by moving the monocular camera 23, which is used as the imaging device 9 for acquiring images used to control the display of the 3D image, using the moving mechanism 24, based on the posture or work content of the work device 5. As a result, the display system 100 can further reduce the number of monocular cameras, and can display a 3D image that makes it easier for the operator to grasp depth perception with a simpler configuration.

[0062] Figure 10 is a sequence diagram illustrating the operation of the construction machine 1 and remote control device 30 shown in Figure 2. The operation shown in Figure 10 is repeated at predetermined control cycles.

[0063] In step S101, the posture detection device 16 of the construction machine 1 detects posture information indicating the posture of the construction machine 1, particularly the posture of the work device 5.

[0064] In step S102, the communication device 18 of the construction machine 1 transmits the posture information detected by the posture detection device 16 to the remote control device 30.

[0065] In step S103, the control device 34 of the remote control device 30 calculates the bucket position based on the attitude information transmitted from the construction machine 1.

[0066] In step S104, the control device 34 of the remote control device 30 determines the content of the work being performed by the work device 5 on the construction machine 1, based on the bucket position calculated in step S103 and the operation information of the operation device 31 acquired in advance.

[0067] In step S105, the control device 34 of the remote control device 30 determines the focal position of the three-dimensional image displayed on the display device 32 based on the bucket position calculated in step S103 or the work content of the work device 5 determined in step S104.

[0068] In step S106, the control device 34 of the remote control device 30 determines the imaging device 9 (first stereo camera 21 or second stereo camera 22, or two monocular cameras 23, etc.) from which the captured image used for controlling the display of the three-dimensional image is acquired, so that the focal position of the three-dimensional image displayed on the display device 32 becomes the focal position determined in step S105.

[0069] In step S107, the communication device 33 of the remote control device 30 transmits a command to the construction machine 1 specifying the imaging device 9 determined in step S106.

[0070] In step S108, the image processing device 17 of the construction machine 1 causes the imaging device 9, which is specified in the command transmitted from the remote control device 30, to acquire an image.

[0071] In step S109, the communication device 18 of the construction machine 1 transmits the image acquired in step S108 to the remote control device 30.

[0072] In step S110, the control device 34 of the remote control device 30 performs display control of the 3D image using the captured image transmitted from the construction machine 1.

[0073] In step S111, the display device 32 of the remote control device 30 displays a three-dimensional image by the display control performed in step S110.

[0074] In this way, the display system 100 of this embodiment can display a three-dimensional image on the display device 32 with a focal position changed according to the posture of the work device 5 or the work being done. As a result, the display system 100 can display an image that makes it easy for the operator to grasp the sense of depth, even if the posture of the work device 5 or the work being done changes moment by moment. Therefore, the display system 100 makes it easier for the operator to intuitively grasp the positional relationship between the work device 5 and the object, thereby preventing a decrease in the work efficiency of the construction machine 1.

[0075] In the display system 100 of this embodiment, the imaging device 9 was configured with multiple stereo cameras 21, 22, three or more monocular cameras 23, or two or more monocular cameras 23 and a movement mechanism 24 in order to change the focal position of the three-dimensional image. However, if the focal position of the three-dimensional image can be changed using the software of the control device 34, the imaging device 9 may be configured with just one stereo camera 21, 22.

[0076] 1...Construction machine, 3...Lower traveling body, 4...Upper slewing body, 5...Working device, 6...Engine, 7...Main pump, 8...Cabin, 9...Imaging device, 10...Boom, 11...Arm, 12...Bucket, 13...Boom cylinder, 14...Arm cylinder, 15...Bucket cylinder, 16...Attitude detection device, 17...Image processing device, 18...Communication device, 21...First stereo camera, 22...Second stereo camera, 23...Monocular camera, 24...Movement mechanism, 30...Remote control device, 31...Operation device, 32...Display device, 33...Communication device, 34...Control device, 34A...Control unit, 34B...Determination unit, 100...Display system

Claims

1. A display system for displaying an object being worked on by a work device of a construction machine, comprising: a control device that performs display control to display a three-dimensional image of the object using an image of the object acquired by an imaging device attached to the construction machine; and a display device that displays the three-dimensional image according to the display control of the control device, wherein the control device changes the focal position of the three-dimensional image based on the posture or work content of the work device, and the display device displays the three-dimensional image with the changed focal position.

2. The construction machine comprises a vehicle body that rotatably supports the work device, and the control device changes the focal position of the three-dimensional image to a position further away from the vehicle body as the tip position of the work device is further away from the vehicle body, characterized in that the display system for the construction machine according to claim 1.

3. The construction machine is a machine that includes a vehicle body that rotatably supports the work device, and the work device performs excavation work in which it excavates an object, and loading work in which the work device loads the excavated object onto another machine, and the control device is characterized in that when the work is excavation work, the focal position of the three-dimensional image is changed to a position further from the vehicle body than when the work is loading work, as described in claim 1.

4. The display system for a construction machine according to claim 1, characterized in that the display device is capable of displaying a two-dimensional image of the object, the control device performs the display control to switch between the three-dimensional image and the two-dimensional image according to the work content, and the display device switches between and displays the three-dimensional image and the two-dimensional image according to the display control of the control device.

5. The display system for a construction machine according to claim 1, wherein the imaging device is composed of a plurality of stereo cameras, and the control device changes the focal position of the three-dimensional image by selecting one stereo camera from the plurality of stereo cameras as the imaging device from which the captured image used for display control of the three-dimensional image is acquired, based on the posture of the work machine or the work content.

6. The construction machine comprises a vehicle body that rotatably supports the work device, the plurality of stereo cameras include a first stereo camera whose working distance, which is the distance from the stereo camera to the object, is a predetermined value, and a second stereo camera whose working distance is longer than the predetermined value, the second stereo camera being mounted on the vehicle body at a higher position than the first stereo camera, the control device selecting the first stereo camera as the imaging device from which the captured image used for display control of the three-dimensional image is acquired when the focal position of the three-dimensional image is set to a position close to the vehicle body based on the posture of the work device or the work content, and selecting the second stereo camera as the imaging device from which the captured image used for display control of the three-dimensional image is acquired when the focal position of the three-dimensional image is set to a position far from the vehicle body based on the posture of the work device or the work content, the display system for the construction machine according to claim 5.

7. The display system for a construction machine according to claim 1, wherein the imaging device is composed of three or more monocular cameras, and the control device changes the focal position of the three-dimensional image by selecting two monocular cameras from the three or more monocular cameras as the imaging device from which the captured images used for display control of the three-dimensional image are acquired, based on the posture of the work device or the work content.

8. The display system for a construction machine according to claim 1, wherein the imaging device comprises two or more monocular cameras and a moving mechanism for moving the monocular cameras so that the distance between them changes in the left-right direction of the construction machine, and the control device changes the focal position of the three-dimensional image by moving the monocular camera with the moving mechanism as the imaging device from which the captured image used for display control of the three-dimensional image is acquired, based on the posture of the work machine or the work content.

9. The construction machine is remotely operated by a remote control device, and the remote control device comprises the control device and the display device, characterized in that the display system for the construction machine according to claim 1.

Images