Remote control device, remote operation system, and remote control method

Abstract

The present invention provides technology for assisting an operator in remotely operating a work machine to work in cooperation with another work machine. This remote control device has a controller that: generates, in a virtual space, a three-dimensional model of a remotely operated work machine; receives a remote operation performed by an operator and directed to the work machine; generates a remote display image for displaying the work machine operating on the basis of the remote operation; generates a virtual space image for displaying the three-dimensional model of the work machine; and simultaneously displays the remote display image and the virtual space image.

Description

Remote Control Device, Remote Operation System, and Remote Control Method

[0001] The present disclosure relates to a remote control device, a remote operation system, and a remote control method.

[0002] With the recent progress in information and communication technology, research and development of technologies for remotely operating working machines such as hydraulic excavators, cranes, and bulldozers has been promoted. In typical remote operation of a working machine, an operating environment similar to that of the cockpit of the working machine is provided in a remote control room. An operator remotely operates the working machine disposed at a construction site while checking an image projected on a display from the cockpit of the working machine (Patent Document 1).

[0003] An image projected on a display in a remote control room may be perceived by an operator differently from the actual view from the cockpit of the actual working machine. In particular, the operator may perceive perspective and depth differently between the image displayed in the remote control room and the view from the cockpit of the actual machine, which can degrade the operator's operation performance. Techniques have been proposed to address such a difference between the image on the display and the actual view.

[0004] Japanese Unexamined Patent Application Publication No. 2016-160741

[0005] When an operator remotely operates a hydraulic excavator, the difference between the image on the display and the actual view makes it difficult for the operator to perform remote operation.

[0006] An object of the present disclosure is to provide a technique for assisting an operator in remotely operating a working machine.

[0007] One aspect of the present disclosure relates to a remote control device having a controller, wherein the controller generates a three-dimensional model of a remotely operated working machine in a virtual space, accepts remote operation of the working machine by an operator, generates a remote display image that displays the working machine operating based on the remote operation, generates a virtual space image that displays the three-dimensional model of the working machine, and displays the remote display image and the virtual space image simultaneously.

[0008] Another aspect of the present disclosure relates to a remote control system comprising a remote control device and an operator interface device, wherein the remote control device has a controller that generates a three-dimensional model of a remotely controlled work machine in a virtual space, receives remote operations from an operator to the work machine, generates a remote display image showing the work machine operating based on the remote operations, generates a virtual space image showing the three-dimensional model of the work machine, and displays the remote display image and the virtual space image simultaneously.

[0009] Another aspect of this disclosure relates to a remote control method in which a computer implements: generating a three-dimensional model of a remotely controlled work machine in a virtual space; receiving remote operations from an operator to the work machine; generating a remote display image showing the work machine operating based on the remote operations; generating a virtual space image showing the three-dimensional model of the work machine; and simultaneously displaying the remote display image and the virtual space image.

[0010] According to this disclosure, it is possible to provide technology to assist operators in remotely controlling work machines and coordinating them with other work machines.

[0011] Figure 1 is a schematic diagram showing a remote control system according to one embodiment of the present disclosure. Figure 2 is a diagram showing a remote display image of a work machine according to one embodiment of the present disclosure. Figure 3 is a diagram showing a virtual space image of a work machine according to one embodiment of the present disclosure. Figure 4 is a block diagram showing the hardware configuration of a remote control device according to one embodiment of the present disclosure. Figure 5 is a flowchart showing the remote control process according to one embodiment of the present disclosure. Figure 6 is an image diagram showing the screen of a setting monitor according to one embodiment of the present disclosure. Figure 7 is an image diagram showing the screen of a setting monitor according to one embodiment of the present disclosure. Figure 8 is an image diagram showing the screen of a setting monitor according to one embodiment of the present disclosure. Figure 9 is an image diagram showing the screen of a setting monitor according to one embodiment of the present disclosure.

[0012] Embodiments of this disclosure will be described below with reference to the drawings.

[0013] The following embodiments disclose remote control technology for work machines.

[0014] [Outline] The remote control system 20 comprises a remote control device 100 and an operator interface device. In the following embodiment, the remote control device 100 displays a remote display image VR1 provided to the operator of the work machine 10 on a display 520 installed in the remote control room 500. The remote control device 100 also displays a virtual space image VR2, which shows the work machine 10 remotely controlled by the operator from an external viewpoint, on the display 520 installed in the remote control room 500. As shown in Figure 1, the work machine 10 remotely controlled by the operator may be, for example, an excavator such as a hydraulic excavator. The view from the driver's seat 12 of the work machine 10 is displayed as the remote display image VR1 on the display 520 installed in the remote control room 500. Note that the remote display image VR1 may be displayed on VR goggles (not shown) worn by the operator, rather than on the display 520 installed in the remote control room 500.

[0015] As shown in the figure, the display 520 is connected to a remote control device 100, which is implemented by a computing device such as a server or a personal computer (PC). The control device 145 of the work machine 10 acquires an image taken by a camera 13 installed on the work machine 10. The control device 145 sends the image to the remote control device 100. The remote control device 100 displays the image on the display 520 as a remote display image VR1.

[0016] The operator operates the remote control handle 50 and pedal 40, which are modeled after the handle and pedals on the actual work machine 10, while viewing the remote display image VR1 projected onto the display 520. The operator operates the work machine 10 as displayed in the remote display image VR1. For example, the operator may perform the work in coordination with other work machines 11, such as a dump truck.

[0017] The cockpit 12 of the actual work machine 10 is equipped with a control monitor 80 for setting and controlling the work machine 10 and displaying the operating status of the work machine 10. The operator can set and control the work machine 10 and check the operating status of each part of the work machine 10 by touching the screen of a setting monitor 95 which is modeled after the control monitor 80, as well as by using the steering wheel 50 and pedals 40 in the remote control room 500.

[0018] In the embodiments described later, the remote control device 100 generates a three-dimensional model of the work machine 10 operated by the operator. The remote control device 100 displays the generated three-dimensional model as a virtual space image VR2 on a display 520 in the remote control room 500. A work machine 11 that works in conjunction with the work machine 10 is also generated as a three-dimensional model in virtual space and displayed on the display 520 in the remote control room 500. The three-dimensional models are created in advance by a CAD (Computer-Aided Design) engineer, designer, etc., and stored in the remote control device 100. The remote display image VR1 and the virtual space image VR2 may be displayed simultaneously on different displays, or they may be displayed simultaneously on a single display with the screen split. Alternatively, any image selected by the operator from the remote display image VR1 and the virtual space image VR2 may be displayed on the display.

[0019] For example, the operator remotely controls the excavator, which is the work machine 10. The operator transports the excavated soil to the bed of a dump truck, which is the work machine 11. The remote control device 100 may display a remote display image VR1, as shown in Figure 2, on a display 520 in the remote control room 500. The illustrated remote display image VR1 projects the view from a camera 13 installed in the driver's seat 12 of the excavator 10. Note that the camera 13 is not limited to being installed in the driver's seat 12, but may be installed on the roof of the driver's seat 12, or elsewhere.

[0020] As shown in the figure, the remote display image VR1 shows the left and right steering wheels 90, the left and right pedals 91, the attachment 92, the arm and boom 93, and the control monitor 80. The remote display image VR1 also shows an image of the dump truck 11 from the rear.

[0021] Furthermore, the surrounding construction site and other views visible from the cockpit 12 are displayed on the remote display image VR1.

[0022] In the following embodiment, the remote control device 100 may display a virtual space image VR2, as shown in Figure 3, on a display 520 in the remote control room 500. In the remote display image VR1 shown in Figure 2, the operator's seat 12 of the work machine 10 and the work machine 11 are projected as seen from the operator's seat 12 of the work machine 10. In the virtual space image VR2 shown in Figure 3, the work machine 10 and the work machine 11 are projected from a position and direction different from the direction seen from the operator's seat 12 of the work machine 10 (in the illustrated example, the view from outside the operator's seat 12 of the work machine 10 in the virtual space). In the virtual space image VR2, the positional relationship between the work machine 10 and the work machine 11 is projected so that the operator can better see it on the display 520.

[0023] The field of view from an external position of the work machine 10 may be set to a field of view from any position by, for example, operating the setting monitor 95. The operator can set the field of view on the setting monitor 95 to a field of view from any position that allows for better visualization of the positional relationship between work machine 10 and work machine 11. The positional relationship between the actual work machine 10 and the actual work machine 11 at the construction site may be determined based on positioning results from any type of positioning sensor, such as a GPS (Global Positioning System) sensor, installed on each actual machine.

[0024] To better visualize the positional relationship between the work machine 10 that the operator is operating and other work machines 11 that work in conjunction with work machine 10, the operator can view a virtual space image VR2 generated from an image taken from any position and direction suitable for the current work, using three-dimensional models of work machines 10 and 11.

[0025] Here, the remote control device 100 may be implemented by a computing device such as a server, personal computer (PC), smartphone, or tablet. For example, it may have a hardware configuration as shown in Figure 4. That is, the remote control device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106 that are interconnected via bus B.

[0026] The programs or instructions that implement the various functions and processes of the remote control device 100 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a USB (Universal Serial Bus) memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may also be downloaded from any external device via a network or the like.

[0027] The storage device 102 is implemented by a hard disk drive or the like, and stores files, data, etc., used to execute the installed program or instruction, along with the program or instruction itself.

[0028] The memory device 103 is implemented using random access memory, static memory, etc., and when a program or instruction is activated, it reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitor storage medium.

[0029] The processor 104 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc., which may consist of one or more processor cores. The processor 104 executes various functions and processes of the remote control device 100 according to data such as programs, instructions, and parameters necessary to execute the programs or instructions stored in the memory device 103.

[0030] The user interface (UI) device 105 may consist of input devices such as a keyboard, mouse, camera, and microphone; output devices such as a display, speaker, headset, and printer; and input / output devices such as a touch panel. The user interface (UI) device 105 provides an interface between the user and the remote control device 100. For example, the user may operate the remote control device 100 by operating a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, mouse, etc.

[0031] The communication device 106 is implemented by various communication circuits that perform wired and / or wireless communication processing with external devices, the Internet, LAN (Local Area Network), cellular networks, and other communication networks.

[0032] However, the hardware configuration described above is merely an example, and the remote control device 100 according to this disclosure may be implemented by any other suitable hardware configuration.

[0033] Furthermore, the pedal 40, handle 50, and display 520 shown in Figure 1 can be collectively referred to as an operator interface device that transmits communication between the remote control device 100 and the operator. The operator interface device according to this disclosure is not necessarily limited to the pedal 40, handle 50, and display 520. Other devices depending on the various forms of the work machine 10 may be used as the operator interface device. The remote control system 20 may consist of the remote control device 100 and the operator interface device.

[0034] [Remote Control Processing] Next, remote control processing according to one embodiment of the present disclosure will be described. This remote control processing is performed by a remote control device 100 implemented by a server, PC, etc. More specifically, a controller 110 implemented by the processor 104 of the remote control device 100 executes remote control processing according to programs, instructions and / or data stored in a non-transitor computer-readable storage medium of a storage device 102 and a memory device 103, etc. (hereinafter collectively referred to as memory 120). Figure 5 is a flowchart showing remote control processing according to one embodiment of the present disclosure.

[0035] In step S1, the controller 110 generates three-dimensional models in virtual space of the work machine 10, which is remotely operated by the operator, and the work machine 11, which works in conjunction with the work machine 10. Specifically, the controller 110 extracts the three-dimensional models of the work machine 10 and the work machine 11 from memory 120 and obtains the positional information (e.g., latitude and longitude, orientation, etc.) of each actual work machine 10 and work machine 11 at the construction site. Then, the controller 110 superimposes the three-dimensional models of the work machine 10 and the work machine 11 onto the three-dimensional model representing the virtual work environment of the construction site, based on the position and orientation of each actual machine, and generates a virtual space image VR2.

[0036] The virtual work environment of the construction site can be modeled in 3D by generating mesh data using point cloud data and distance information of the construction site measured by, for example, 3D distance sensors or stereo cameras installed on the work machine 10 or other work machines 11. This makes it easier to understand not only the positional relationship between work machines 10 and 11, but also the positional relationship between the attachment 92 of work machine 10 and the surrounding ground, etc., and helps prevent damage caused by contact between work machine 10 and the ground, etc.

[0037] In step S2, the controller 110 accepts remote control of the work machine 10 from the operator. Specifically, the operator performs remote operations to carry out various tasks, such as transporting the soil excavated by the work machine 10 to the loading platform of the work machine 11, while checking the remote display image generated in step S1. For example, when the operator operates the pedals 40, handle 50, etc., in the remote control environment installed in the remote control room 500, the controller 110 transmits an operation signal indicating the operator's operation to the actual work machine 10. The controller 110 then operates the 3D models of the work machine 10 and work machine 11 in the virtual space according to the operation.

[0038] The controller 110 may accept operator input to the setting monitor 95. The operator can specify any viewpoint on the setting monitor 95 that allows for better visualization of the relative positions of the work machines 10 and 11. This viewpoint is any position and direction different from the viewpoint of the operator in the cockpit 12. For example, it could be the position of the work machine 10 as seen from the outside. As shown in Figure 6, the operator touches the screen of the setting monitor 95 with their finger 96. When the operator touches the item "Display Settings" from the main menu screen, the controller 110 detects that the item "Display Settings" has been selected by the operator and transitions to the display settings screen on the setting monitor 95.

[0039] As shown in Figure 7, the operator touches the screen of the setting monitor 95 with their finger 96. When the operator selects the item "Viewpoint Setting" on the display setting screen, the controller 110 detects that the item "Viewpoint Setting" has been selected by the operator.

[0040] As shown in Figure 8, the controller 110 displays the view image VW1 currently displayed to the operator and the positional relationship image RP1 showing the positional relationship between the work machine 10 and the work machine 11 on the setting monitor 95. The positional relationship image RP1 is an image viewed from an arbitrary position and direction. An arbitrary position and direction is a position and direction different from the viewpoint of the operator in the cockpit 12, and is displayed as a viewpoint pointer VP below the setting monitor 95. Here, the dashed line drawn from the viewpoint pointer VP represents the view direction visible from that position.

[0041] The operator changes the viewpoint by touching the viewpoint pointer VP on the setting monitor 95 with their finger 96. The controller 110 displays the modified view image VW2 on the setting monitor 95 based on the viewpoint pointer VP changed by the operator (see Figure 9).

[0042] Touch operations may include various operations such as swiping, long-pressing, and pinching, which are used on smartphones, tablets, etc. The controller 110 may determine what operation the operator performed according to the detected operation. Alternatively, the operation may be performed by rotating two fingers while they are in contact with the monitor. In addition, instead of touch operations, the viewpoint may be set by selecting viewpoint selection buttons such as "overhead view," "XY plane," and "side view" displayed on the setting monitor 95.

[0043] The viewpoint of the virtual space image VR2 is not limited to being set using the setting monitor 95. For example, it may be set by operating various input devices such as a mouse, switch, or keyboard connected to the remote control device 100.

[0044] In step S3, the controller 110 controls images to display the working machine 10 and working machine 11 that operate in accordance with remote operation from a viewpoint different from the viewpoint displayed to the operator. Specifically, when receiving a request to change the viewpoint from the operator via the setting monitor 95, the controller 110 projects the virtual space image VR2 from the changed viewpoint onto the display 520 in the remote control room 500.

[0045] For example, when the viewpoint is changed to the left side of the working machine 10 as shown in FIG. 9, the controller 110 displays the virtual space image VR2 as shown in FIG. 3 on the display 520 in the remote control room 500. According to the virtual space image VR2, compared with the remote display image VR1 as shown in FIG. 2, the sense of distance between the working machine 10 and the working machine 11 is visually expressed. The operator can easily visually recognize how far the working machine 10 should be moved to transfer the sediment carried by the working machine 10 onto the loading platform of the working machine 11. In the virtual space image VR2, the viewing direction from the changed viewpoint pointer VP may be drawn by a broken line.

[0046] The controller 110 may display the positional relationship in an enlarged manner. For example, if the working machine 10 includes an attachment 92 such as a bucket that holds excavated sediment, and the working machine 11 includes a loading platform for carrying an object such as sediment transferred by the attachment 92, the operator needs to move the attachment 92 to above the loading platform without contacting the loading platform, and drop the sediment onto the loading platform.

[0047] An operator mainly needs to pay close attention to the positional relationship between the attachment 92 and the loading platform, and displaying the positional relationship in an enlarged manner is useful for the operator. The controller 110 may display an enlarged view of the positional relationship between the attachment 92 and the loading platform on the display 520 in the remote control room 500. The enlargement and reduction of the virtual space image VR2 may be set by an operator's operation on the setting monitor 95. The scaling of the virtual space image VR2, and the translational and rotational movement of the camera viewpoint may be implemented by pinch-in / pinch-out using two fingers, a rotating motion with two fingers in contact with the setting monitor 95, a swipe operation, or the like. The viewpoint of the virtual space image VR2 is not limited to being set via the setting monitor 95, and may be set, for example, by operating various input devices such as a mouse, a switch, and a keyboard connected to the remote control device 100.

[0048] According to the remote control process described above, the operator can more favorably visually recognize the positional relationship between the operating work machine 10 and another work machine 11 that operates in cooperation with the work machine 10. The operator can reliably remotely control the work machine 10 while checking the virtual space image VR2 generated for any viewpoint suitable for the work.

[0049] According to the remote control process described above, the operator can similarly more favorably visually recognize the positional relationship between the work machine 10 and a construction site. The operator can reliably remotely control the work machine 10 while checking the virtual space image VR2 generated for any viewpoint suitable for the work.

[0050] Hitherto, the embodiments of the present disclosure have been described in detail. However, the present disclosure is not limited to the specific embodiments described above, and various modifications and changes can be made within the scope of the gist of the present disclosure described in the claims.

[0051] 10, 11: Work machine; 20: Remote control system; 40: Pedal; 50: Handle; 95: Setting monitor; 100: Remote control device

Claims

1. A remote control device having a controller that generates a three-dimensional model of a remotely operated work machine in a virtual space, receives remote operation of the work machine by an operator, generates a remote display image that shows the work machine operating based on the remote operation, generates a virtual space image that shows the three-dimensional model of the work machine, and displays the remote display image and the virtual space image simultaneously.

2. The remote control device according to claim 1, wherein the remote display image and the virtual space image display images viewed from different locations.

3. The remote control device according to claim 1 or 2, wherein the work machine is a first work machine, a three-dimensional model of a second work machine is generated in a virtual space, and the three-dimensional model of the second work machine is displayed in the virtual space image.

4. The remote control device according to claim 3, wherein the first work machine comprises an attachment, the second work machine comprises a platform for transporting an object moved by the attachment, and controlling the virtual space image visualizes the positional relationship between the attachment and the platform.

5. The remote control device according to claim 4, wherein controlling the virtual space image enlarges and displays the positional relationship.

6. The remote control device according to claim 3, wherein controlling the virtual space image provides an overhead view of the three-dimensional model of the work machine.

7. A remote control device according to claim 1, which generates a three-dimensional model of a construction site in a virtual space and displays the three-dimensional model of the construction site in the virtual space image.

8. A remote control system comprising: a remote control device and an operator interface device, wherein the remote control device generates a three-dimensional model of a work machine to be remotely controlled in a virtual space, receives remote control of the work machine by an operator, generates a remote display image showing the work machine operating based on the remote control, generates a virtual space image showing the three-dimensional model of the work machine, and displays the remote display image and the virtual space image simultaneously.

9. A remote control method in which a computer implements: generating a three-dimensional model of a remotely controlled work machine in a virtual space; receiving remote control of the work machine by an operator; generating a remote display image that shows the work machine operating based on the remote control; generating a virtual space image that shows the three-dimensional model of the work machine; and simultaneously displaying the remote display image and the virtual space image.

Images