Automatic operation system

Abstract

This automatic operation system (1) has an information processing device (40). A work machine (10) is capable of automatic operation. The information processing device (40) is connected to the work machine (10) so as to be able to communicate. The information processing device (40) comprises a display unit (42), an input unit (41), and a control unit (43). The display unit (42) displays information related to the work machine (10). The input unit (41) receives an operation by a worker. The control unit (43) controls a display image (G) to be displayed on the display unit (42) and information related to input into the input unit (41). The control unit (43) executes an automatic operation process and a remote control process. The automatic operation process displays information related to automatic operation on the display unit (42). In the automatic operation process, the input unit (41) receives input of the information related to automatic operation. The remote control process remotely controls the work machine (10) on the basis of the operation received by the input unit (41).

Description

Automatic driving system

[0001] The present invention relates to an automatic driving system.

[0002] For example, Patent Document 1 discloses a work machine that performs automatic driving.

[0003] Japanese Patent Application Laid-Open No. 2018-193704

[0004] During the automatic driving of the work machine, the vehicle body of the work machine may become unstable, such as floating. While the automatic driving is executed by sending basic commands from a tablet held by the operator, to return from such an unstable posture, the work machine needs to be operated by another operating device different from the tablet (a remote operating device, a remote control device, an operating device in the cab of the work machine). In this case, there was a problem that it took time for the operator to return the posture of the work machine.

[0005] An object of the present invention is to provide an automatic driving system that can easily return a work machine that has become unstable during automatic driving.

[0006] An automatic driving system according to one aspect of the present invention includes an information processing device. The work machine is capable of performing automatic driving. The information processing device is communicably connected to the work machine. The information processing device has a display unit, an input unit, and a control unit. The display unit displays information about the work machine. The input unit receives an operation of the operator. The control unit controls a display image to be displayed on the display unit and information input to the input unit. The control unit executes an automatic driving process and a remote control process. In the automatic driving process, the control unit displays information about the automatic driving on the display unit. In the automatic driving process, the control unit receives an input of information about the automatic driving at the input unit. Then, the control unit transmits a command related to automatic driving to the work machine. On the other hand, in the remote control process, the control unit transmits a command related to remote control to the work machine and remotely controls the work machine based on an operation related to the remote control received by the input unit.

[0007] The above-described automated driving system allows for easy recovery of work machinery that becomes unstable during automated driving via an information processing device.

[0008] Figure 1 shows an automated driving system including a work machine and an information processing device. Figure 2 is a block diagram of the automated driving system shown in Figure 1. Figure 3 shows the automated driving mode screen displayed on the display unit shown in Figure 2. Figure 4 shows the vehicle lift detection screen shown on the display unit shown in Figure 3. Figure 5 shows the normal operation screen shown on the display unit shown in Figure 3. Figure 6 is a command signal table used for the driving operation unit on the normal operation screen of Figure 5. Figure 7 shows the driving operation screen shown on the display unit shown in Figure 3. Figure 8 shows the attachment operation screen shown on the display unit shown in Figure 3. Figure 9 is an explanatory diagram of the selection of attachment operation on the attachment operation screen of Figure 8. Figure 10 shows the attachment operation screen when the work machine shown on the display unit shown in Figure 8 is in an unstable position. Figure 11 shows a modified example of the normal operation screen of Figure 5.

[0009] An automated driving system 1 according to one embodiment of the present invention will be described with reference to the drawings.

[0010] As shown in Figure 1, the automatic driving system 1 is a system related to the automatic driving of the work machine 10. The automatic driving system 1 is a system that restores the work machine 10 when it has become unstable. The automatic driving system 1 comprises the work machine 10, an information processing device 40, and a detection unit 31 (Figure 2). Note that the automatic driving system 1 does not include the work machine 10, but may be a system that can control the work machine 10.

[0011] The work machine 10 is a machine that performs work. The work machine 10 may be a construction machine that performs construction work, or a material handling machine that performs material handling work. The work machine 10 may be, for example, a shovel or a crane. The work machine 10 may be a bulldozer or a wheel loader. The following describes the case where the work machine 10 is a shovel. The work machine 10 is configured to be automatically operated. The work machine 10 may also operate in response to the operation of a worker (operator) without the use of automatic control. The work machine 10 may be operated by a worker (operator) in the driver's cab 13a (described later). The work machine 10 is configured to be remotely operated from outside the work machine 10. The number of work machines 10 included in the automatic driving system 1 may be one or multiple. The work machine 10 has a machine body 10a, an attachment 15, a drive control unit 17, and an actuator 21.

[0012] The machine body 10a is the main body portion of the work machine 10. The machine body 10a includes a lower traveling body 11, a slewing device 12, and an upper slewing body 13. The lower traveling body 11 is capable of traveling on a traveling surface (such as the ground). The lower traveling body 11 may have crawlers or wheels. As shown in Figure 1, in this embodiment, the lower traveling body 11 includes a traveling body base 11a, a left traveling body 11bL as one of the drive units, and a right traveling body 11bR as one of the drive units. The traveling body base 11a is a frame (structure) that supports the upper slewing body 13 via the slewing device 12 (described later). The traveling body base 11a connects the left traveling body 11bL and the right traveling body 11bR. The left traveling body 11bL is located on the left side of the lower traveling body 11. The right traveling body 11bR is located on the right side of the lower traveling body 11. The left travel body 11bL is capable of traveling in both forward and reverse directions (the same applies to the right travel body 11bR). The "forward direction" of the left travel body 11bL is one direction along the longitudinal direction of the left travel body 11bL (the same applies to the right travel body 11bR). The "reverse direction" of the left travel body 11bL is the direction opposite to the forward direction of the left travel body 11bL (the same applies to the right travel body 11bR). The right travel body 11bR is attached to the travel body base 11a on the side opposite to the side to which the left travel body 11bL is attached. The slewing device 12 is a device (e.g., a slewing bearing) that supports the upper slewing body 13 so that it can rotatably move relative to the lower travel body 11. The upper slewing body 13 is rotatably supported by the lower travel body 11. The upper slewing body 13 has an operator's cab 13a. The operator's cab 13a is the part from which an operator can operate the work machine 10. Furthermore, when the work machine 10 operates in response to the operator's operation, the work machine 10 may be operated (operated while on board) by the operator inside the driver's cab 13a, or it may be remotely operated from outside the work machine 10.

[0013] Attachment 15 is a work device attached to the machine body 10a. Attachment 15 is the part that performs work and its components include, for example, a boom 15a, an arm 15b, and a tip attachment 15c. The boom 15a is rotatably attached to the upper slewing body 13. The arm 15b is rotatably attached to the boom 15a.

[0014] The tip attachment 15c is provided at the tip of the attachment 15. The tip attachment 15c is rotatably attached to the arm 15b. The tip attachment 15c performs work on the workpiece. The tip attachment 15c may be a bucket capable of scooping and excavating the workpiece. The tip attachment 15c may be equipped with a device for gripping the workpiece (grapple, nibbler, rotating fork, etc.), a device for crushing the workpiece (breaker, etc.), or a magnet for attracting metal workpieces.

[0015] The work object is the object that the work machine 10 operates on. The work object may be soil, rock, magnetic material (such as metal), resin, waste, wood (such as logs), or structure (such as blocks). If the work object is soil, it may be in the form of soil, granules, chips, powder, etc.

[0016] The work machine 10 may also have a dozer as a work device for performing work. The dozer is a work device attached to the lower traveling body 11 (for example, the base portion 11a of the traveling body). The dozer has, for example, a plate-shaped member (for example, a dozer blade) that extends in the width direction and the vertical direction of the lower traveling body 11. The dozer may also be movable in the vertical direction relative to the lower traveling body 11.

[0017] The actuator 21 is a device that moves the work machine 10. The actuator 21 may have a hydraulic actuator that is driven by hydraulic pressure, or an electric actuator that is driven by electricity. The actuator 21 may have a motor that rotates, or a cylinder that extends and retracts (extendable cylinder).

[0018] The actuator 21 includes a travel motor 26, a slewing motor 21a, a boom cylinder 21b, an arm cylinder 21c, and a tip attachment cylinder 21d. The travel motor 26 drives the lower travel body 11. For example, as shown in Figure 1, if the lower travel body 11 has a left travel body 11bL and a right travel body 11bR, a motor is provided to drive the lower travel body 11 and a motor is provided to drive the right travel body 11bR. The travel motor 26 may be a hydraulic motor or an electric motor (the same applies to the slewing motor 21a). The slewing motor 21a rotates the upper slewing body 13 relative to the lower travel body 11. The boom cylinder 21b raises and lowers the boom 15a relative to the upper slewing body 13. The boom cylinder 21b is, for example, a hydraulic cylinder (the same applies to the arm cylinder 21c and the tip attachment cylinder 21d). The arm cylinder 21c rotates the arm 15b relative to the boom 15a. The tip attachment cylinder 21d rotates the tip attachment 15c relative to the arm 15b. If the tip attachment 15c itself is drivable, such as a device for gripping objects, an actuator 21 for driving the tip attachment 15c may be provided.

[0019] The drive control unit 17 (Figure 2) controls the actuator 21 that moves the work machine 10. The drive control unit 17 may have a hydraulic circuit that controls a hydraulic actuator that operates by hydraulic pressure. The drive control unit 17 may also have an electrical circuit that controls an electric actuator that operates by electric power.

[0020] The drive control unit 17 (Figure 2) controls the travel motor 26 that drives the lower travel body 11. The drive control unit 17 controls the slewing motor 21a that slewing the upper slewing body 13 relative to the lower travel body 11. The drive control unit 17 controls the boom cylinder 21b that rotates (raises and lowers) the boom 15a relative to the upper slewing body 13. The drive control unit 17 controls the arm cylinder 21c that rotates the arm 15b relative to the boom 15a. The drive control unit 17 controls the tip attachment cylinder 21d that rotates the tip attachment 15c relative to the arm 15b.

[0021] The detection unit 31 (Figure 2) detects various states. Part or all of the detection unit 31 may be mounted on the work machine 10 or located outside the work machine 10. The same applies to the information processing device 40, which will be described later, in that it may be mounted on the work machine 10 or located outside the work machine 10.

[0022] As shown in Figure 2, the detection unit 31 includes a position detection unit 311, a direction detection unit 312, an imaging device 313, and an attitude detection unit 315. The position detection unit 311 detects the position of the object to be measured. The position detection unit 311 detects the position of a specific part of the work machine 10 (Figure 1). For example, the position detection unit 311 may detect the position of a specific part of the upper rotating body 13 (Figure 1), or it may detect the position of a specific part of the attachment 15 (Figure 1). The position detection unit 311 may include a device that detects position using electromagnetic waves (light, radio waves, etc.). The position detection unit 311 may also include a device that uses a satellite positioning system, for example, a device that uses GNSS (Global Navigation Satellite System). The position detection unit 311 may have a device for detecting position without using satellites, a device for detecting position using a ground-based transmitter and receiver, or a device for detecting position using the reflection of light (e.g., laser light) (e.g., a total station). The position detection unit 311 may calculate the position of the object to be measured based on position information detected by multiple types of devices.

[0023] The direction detection unit 312 detects the direction (orientation, posture) of the object to be measured. The direction detection unit 312 detects the direction of a specific part of the work machine 10 (Figure 1). For example, the direction detection unit 312 may detect the direction of a specific part of the upper rotating body 13, or it may detect the direction of a specific part of the attachment 15 (Figure 1). The direction detection unit 312 may have a device that uses the Earth's magnetic field to detect the orientation of the object to be measured. The direction detection unit 312 may also detect the direction of the object to be measured based on the positions of multiple parts of the object to be measured relative to the work site (for example, positions detected by the position detection unit 311).

[0024] The imaging device 313 images the object to be imaged. The imaging device 313 may image part or all of the work machine 10 (Figure 1), or it may image objects around the work machine 10. The imaging device 313 may detect two-dimensional information (two-dimensional image), or it may detect three-dimensional information (three-dimensional image, distance image) that includes depth information. The imaging device 313 may be passive or active. Specifically, the imaging device 313 may have a camera (monocular camera) that detects two-dimensional information. The imaging device 313 may have a stereo camera that detects three-dimensional information. The imaging device 313 may detect three-dimensional information of the object to be imaged by irradiating the object with waves such as electromagnetic waves and detecting the reflected waves. The imaging device 313 may have a TOF (Time Of Flight) sensor that detects distance based on the time from wave irradiation to the return of the reflected wave, or it may have a sensor that detects distance based on the frequency of the reflected wave. The imaging device 313 may have a device for detecting three-dimensional information using light (e.g., laser light), for example, LiDAR (Light Detection and Ranging). The imaging device 313 may also have a device for detecting three-dimensional information using radio waves (e.g., millimeter-wave radar).

[0025] There may be only one imaging device 313, or there may be multiple imaging devices 313. If multiple imaging devices 313 are provided, the types of imaging devices 313 (type of imaging method, two-dimensional or three-dimensional, etc.) may be the same or different. The imaging device 313 may detect information about the object to be imaged (e.g., three-dimensional information) by combining multiple types of information (e.g., two-dimensional information and three-dimensional information).

[0026] The posture detection unit 315 detects the posture of the work machine 10 (Figure 1). The posture detection unit 315 may also detect the position and orientation of the work machine 10 relative to the work site. The posture detection unit 315 may also detect the position and orientation of a reference position of the work machine 10 relative to the work site. The reference position of the work machine 10 is, for example, a specific position of the upper slewing body 13 (Figure 1) or the lower traveling body 11 (Figure 1). The reference position of the work machine 10 may be the attachment point (boom foot) of the boom 15a (Figure 1) to the upper slewing body 13, or a specific position on the pivot axis of the upper slewing body 13 relative to the lower traveling body 11. The posture detection unit 315 may also detect the inclination of the work machine 10 with respect to the horizontal plane. The posture detection unit 315 may also detect information (angle, angular velocity, angular acceleration, etc.) of the rotation of the upper slewing body 13 relative to the lower traveling body 11. The attitude detection unit 315 may detect information about the rotation of the boom 15a relative to the upper slewing body 13 (angle, angular velocity, angular acceleration, etc.). The attitude detection unit 315 may also detect information about the rotation of the arm 15b (Figure 1) relative to the boom 15a. The attitude detection unit 315 may also detect information about the rotation of the tip attachment 15c (Figure 1) relative to the arm 15b. Note that the attitude detection unit 315 may be an IMU (internal measurement unit).

[0027] The information processing device 40 is a device that processes information related to the work machine 10 (Figure 1). The information processing device 40 is a device that processes information related to the automatic operation of the work machine 10. The information processing device 40 has a function to remotely control the work machine 10 which is capable of automatic operation. The information processing device 40 may also have a function to support the management of the work machine 10 which is capable of automatic operation. The information processing device 40 is connected to the work machine 10 in a communicative manner. Hereinafter, a communicative connection will also be simply referred to as "connection". The information processing device 40 may have various information processing devices. The functions of the information processing device 40 may be distributed and arranged across multiple information processing devices (a distributed system may be configured). Specifically, for example, the information processing device 40 has a server 40a (computer) and an information terminal 40b. The server 40a and the information terminal 40b may be connected by wireless communication or by wired communication. For example, communication may be performed by means of communication such as a mobile phone line, optical fiber line, wireless LAN (Local Area Network), or wired LAN.

[0028] For example, server 40a is a computer (controller) that performs information input / output, calculations (processing), and information storage. For example, the functions of server 40a are realized by the execution of a program stored in the storage unit of server 40a by the calculation unit. Server 40a is located outside the work machine 10 (Figure 1) and the information terminal 40b. Server 40a is connected to the work machine 10. Information terminal 40b is connected to the work machine 10. Information terminal 40b may be connected to the work machine 10 via server 40a, or it may be connected to the work machine 10 without going through server 40a.

[0029] The information terminal 40b is a computer capable of inputting and outputting information. The information terminal 40b may or may not be portable. As shown in Figure 1, in this embodiment, the information terminal 40b is a tablet, but it may also be a smartphone or a personal computer. The information terminal 40b may also be used in conjunction with the server 40a. The information terminal 40b may be located outside the work machine 10 (Figure 1) or inside the work machine 10 (for example, it may be brought in).

[0030] As shown in Figure 2, the information processing device 40 includes an input unit 41, a display unit 42, and a control unit 43. The input unit 41 is for inputting information (an input device). The input unit 41 accepts operations from the operator. For example, the input unit 41 is operated by the operator and outputs a signal corresponding to the operation. The input unit 41 inputs information to the control unit 43. In this embodiment, the input unit 41 is a touch panel provided on the information terminal 40b (Figure 1). In this case, the display unit 42 can be described as a touch panel type display that also functions as the input unit 41. The input unit 41 may be, for example, a mouse, a keyboard, a device that inputs information based on the operator's gaze, or a device that inputs information based on the position of the operator's fingers in space. The input unit 41 may also have a device for voice input (specifically, a microphone). The input unit 41 may be provided on a smartphone or on a personal computer. Information related to autonomous driving is input to the input unit 41 in response to the operator's operations. The input unit 41 receives information for remotely controlling the work machine 10 (Figure 1) in response to the operator's actions.

[0031] The display unit 42 is a device that outputs information. The display unit 42 displays information related to the work machine 10 (Figure 1). The display unit 42 displays information related to automatic operation. The display unit 42 outputs information based on signals output from the control unit 43. As shown in Figure 1, in this embodiment, the display unit 42 is a monitor (display) provided on the information terminal 40b, but it may be provided on a smartphone or a personal computer. The display unit 42 may have a projection device that projects onto an object such as the ground. The display unit 42 may be equipped with a light emitter (light). The display unit 42 may be equipped with a device that utilizes VR (Virtual Reality) technology (VR device) or a device that utilizes AR (Augmented Reality) technology (AR device). The display unit 42 may change at least one of the hue, density (transparency), brightness, and saturation of the light it outputs. The image displayed by the display unit 42 (the image that the control unit 43 causes the display unit 42 to display) is called the display image G. The display image G includes shapes, patterns, colors, characters, symbols, etc. The display image G includes a GUI (Graphical User Interface). The information terminal 40b may also have a part that performs output other than display (output unit), for example, an audio output unit and a vibration output unit.

[0032] The control unit 43 is a computer (controller) that performs signal input / output, calculations (processing), and information storage. For example, the functions of the control unit 43 are realized by the execution of a program stored in the memory unit of the control unit 43 by the calculation unit. The control unit 43 controls information related to the display unit 42, for example, by causing the display unit 42 to display information. In the following description, causing the display unit 42 to display information means that each control unit inputs a command signal related to the display information to the display unit 42. The control unit 43 controls information related to the input of the input unit 41 (information input to the input unit 41), for example, by processing the information input to the input unit 41. The control unit 43 may also perform other controls, such as information communication control. The control unit 43 may be provided on, for example, a server 40a, or on an information terminal 40b, or its functions may be distributed between the server 40a and the information terminal 40b. The control unit 43 includes an automatic driving control unit 431, a remote control control unit 432, a mode switching control unit 433, and an ambient recognition unit 434.

[0033] The automatic driving control unit 431 causes the display unit 42 to display information related to automatic driving. The automatic driving control unit 431 causes the input unit 41 to accept input related to automatic driving. The automatic driving control unit 431 transmits commands related to automatic driving to the work machine 10. An example of the display on the display unit 42 in automatic driving mode, which displays information for automatic driving processing, will be described later. The remote control control unit 432 remotely operates the work machine 10 (Figure 1) based on the operation received by the input unit 41. Specifically, the remote control control unit 432 transmits commands related to remote operation to the work machine 10. The mode switching control unit 433 switches between automatic driving mode and remote control mode. An example of the display on the display unit 42 in remote control mode, which displays information for remote control processing, will be described later. The surrounding recognition unit 434 acquires (recognizes) the working machine 10, which is performing automatic driving or remote control, and the surrounding conditions of the working machine 10, based on the information acquired from the detection unit 31. When acquiring information about the work machine 10 and its surroundings, the surrounding recognition unit 434 may combine the information from the detection unit 31 with map information of the work site where the work machine 10 is located.

[0034] Furthermore, the operating performance of the work machine 10 (Figure 1) when remotely operated may be limited compared to the operating performance when automatic operation is performed. For example, even when the same command signal is received, the operating speed of the work machine 10 may be set slower in remote operation mode than in automatic operation mode. Also, for example, an upper limit may be set on the operating speed in remote operation mode. For example, the operation of the work machine 10 when remotely operated may be slower than the operation of the work machine 10 when operated by the operating lever of the operating device in the operator's cab 13a or an operating device that mimics the operating device in the operator's cab 13a.

[0035] Thus, in this embodiment, the operator can control both the automatic operation and remote operation of the work machine 10 using the information terminal 40b. Therefore, compared to a case where the information terminal 40b is used exclusively for automatic operation and other operating devices are used for remote operation, the operator's operational burden can be reduced. In particular, as will be described later, even if the posture of the work machine 10 becomes unstable during automatic operation, the operator can switch to remote operation using the information terminal 40b and restore the posture of the work machine 10. Note that the functions of the server 40a may also be incorporated into the information terminal 40b.

[0036] (Automatic Operation of the Work Machine 10) The work machine 10 (Figure 1) is configured to be able to operate automatically based on a work plan. The work plan is information about the objectives of the work of the work machine 10 and is set in advance (before automatic operation) in the information processing device 40. The work plan may also be set by the automatic operation control unit 431. The work plan may include information about the target route for the work machine 10's travel. The work plan may also include information about the target range (e.g., target acquisition range, target release range) in which the tip attachment 15c (Figure 1) will perform its work.

[0037] (Target path, target trajectory) The work plan may include information on the target path (also called the target trajectory) of a specific part of the attachment 15 (Figure 1). The specific part of the attachment 15 may be, for example, the tip of the tip attachment 15c (Figure 1), or the base end of the tip attachment 15c (the tip of the arm 15b (Figure 1)). The specific part of the attachment 15 may be set in only one location, or it may be set in multiple locations. The above "target path" is information that includes, for example, information on the positions (coordinates) of multiple target points and information on the order of each target point. The work plan may also include information on the target trajectory of a specific part of the attachment 15. The above "target trajectory" is information that adds time information to the information on the target path.

[0038] (Cycles, Work Phases) The work plan may include information for having the work machine 10 (Figure 1) perform a certain task (e.g., loading work, lifting magnet work) in multiple cycles. One cycle may include multiple work phases. Specific examples of work phases are as follows. Here, we will explain a specific example of work phases when the work machine 10 automatically performs the task of capturing a work object and moving the captured work object to a predetermined position. In this case, the work phases include a capture phase (e.g., excavation phase), a lifting and rotating phase (soil removal and rotating phase), a release phase (soil removal phase), and a return and rotating phase. The capture phase is the phase in which the tip attachment 15c shown in Figure 1 captures the work object within the target capture range (e.g., excavating soil). For example, the "target capture range" is set to a place where the work objects are collected (e.g., a pile of soil). The lifting and turning phase is the phase in which the tip attachment 15c moves (turns) from the target acquisition range to the target release range while the tip attachment 15c has acquired the work object. The release phase is the phase in which the tip attachment 15c releases the work object in the target release range (for example, by removing soil). The above "target release range" is set to a predetermined range on the bed of a transport vehicle (such as a dump truck). The return and turning phase is the phase in which the tip attachment 15c moves from the target release range to the target acquisition range. For example, in automated driving, the acquisition phase, the lifting and turning phase, the release phase, and the return and turning phase are repeated in a series of work phases (one cycle of automated driving work).

[0039] (Teaching) At least a portion of the work plan may be set by teaching, or by other methods (e.g., numerical input). Information set by teaching is called teaching data. Teaching data is set when an operator operates the work machine 10 (Figure 1). Specifically, for example, the operator may ride on the work machine 10 and operate it, or the operator may remotely operate the work machine 10. For example, the operator may operate the work machine 10 to place a specific part of the attachment 15 (Figure 1) at a location (path, range, etc.) that they want to set as teaching data. Then, based on the position where the specific part of the attachment 15 is placed, the teaching data is set in the information processing device 40. For example, the operator may operate the work machine 10 to place a specific part of the attachment 15 at a specific position within a range that they want to set as a target range (target acquisition range or target release range). The position where a specific part of the attachment 15 is positioned is calculated based on the detection result of the posture detection unit 315, which detects the posture of the work machine 10. Then, the target range is set based on the position where the specific part of the attachment 15 is positioned. For example, the operator moves the specific part of the attachment 15 along the path that they want to set as the target path by operating the work machine 10. For example, the operator moves the specific part of the attachment 15 along the path that they want to set as the target trajectory at the speed they want to set as the target trajectory by operating the work machine 10. Then, the information processing device 40 sets the path (trajectory) along which the specific part of the attachment 15 has moved as the target path (trajectory).

[0040] (Operation) The operation of the information processing device 40, the automatic driving information processing method, and the automatic driving information processing program will be described below. The following will mainly describe the display image G that the control unit 43 displays on the display unit 42. The display image G shown in Figure 3 changes in various ways depending on the situation. The display image G has various parts (elements such as various selection units, various display units, and various setting units). The presence or absence of each part of the display image G, the manner of display, and whether or not it can be selected or operated by the input unit 41 (active or inactive) will change in various ways depending on the situation.

[0041] (Automatic Driving Mode) First, the display image G that the control unit 43 (specifically the automatic driving control unit 431) displays on the display unit 42 in automatic driving mode will be explained. Figure 3 is a diagram showing an example of the display image G in automatic driving mode when the work machine 10 (Figure 1) is in automatic driving mode. In automatic driving mode, the display image G displays information for performing automatic driving processing. For example, in automatic driving mode, the display image G displays a screen that allows the operator to set up automatic driving. Specifically, for example, in automatic driving mode, the display image G displays a screen that allows the operator to select the type of work to be performed in automatic driving mode. Also specifically, for example, in automatic driving mode, the display image G displays a screen that allows the operator to check and modify the settings set for automatic driving. Also specifically, for example, in automatic driving mode, the display image G displays a screen that allows the operator to start, interrupt, and end automatic driving. Also specifically, for example, in automatic driving mode, the display image G displays a screen that shows the status of the work machine 10 during automatic driving. Furthermore, specifically, for example, in the automatic driving mode, display image G shows a screen indicating the work results of the automatic operation performed by the work machine 10.

[0042] In Figure 3, at the far left of the display image G, there is a button to transition to the HOME screen, and buttons for loading, lifting magnet, and adding more tasks as types of work in automatic operation. At the top right, there are tabs for the names of the connected machines 1 and 2. Below that, there are buttons for teaching, loading, trajectory confirmation, position adjustment (loading position adjustment), and settings. In the loading operation settings, the status can be displayed as automatic (automatic operation), remote (remote manual), lock (hydraulic lock), and stop (emergency stop). On the left side of the plan view model image of the hydraulic excavator, a menu button is shown, and on the right side, an image showing the communication status is displayed. The same applies to the other figures described later.

[0043] On such a screen, a mode switching section A is displayed in the display image G. In the automatic driving mode, the mode switching section A is a part that allows an operator to select a switch from the automatic driving mode to the remote operation mode. As shown in the example of FIG. 3, in the automatic driving mode, the mode switching section A may be provided with an indication (e.g., characters) indicating a switch to the remote operation mode. Note that the heavy machine operation mode in the display of FIG. 3 corresponds to the above-mentioned remote operation mode. For example, when the operator selects the mode switching section A, the control unit 43 (specifically, the mode switching control unit 433) switches the user interface of the display unit 42 from the automatic driving mode to the remote operation mode. Thus, the mode switching section A is a switching reception area in the display image G for the input unit 41 to receive a switch between the automatic driving mode and the remote operation mode.

[0044] This "selection" is a selection by the operator operating the input unit 41. Specifically, for example, when the input unit 41 is a touch panel, the selection is that the type of work to be performed in automatic driving is touched. When the input unit 41 is a mouse or a keyboard, the selection may be that an operation (such as clicking or pressing a predetermined key) for determining the selection is performed with the cursor placed on the type of work to be performed in automatic driving. Further, the selection may be that a specific mouse operation or key operation (such as a shortcut key operation) for selecting the type of work to be performed in automatic driving is performed. Hereinafter, the selection by the operator operating the input unit 41 is also simply referred to as "selection" (the same applies to "selection" and "setting" in parts other than the mode switching section A).

[0045] Figure 4 shows the vehicle lift detection screen in automatic driving mode. The vehicle lift detection screen is displayed when vehicle lift, which is an unstable posture for the work machine 10 (Figure 1), is detected and an emergency stop is performed. For example, if the control unit 43 (specifically the surrounding recognition unit 434) determines that vehicle lift has occurred, the display unit 42 displays the vehicle lift detection screen (vehicle lift detection image). For example, the conditions for determining that vehicle lift has occurred include the tilt of the work machine 10 relative to the ground detected by the posture detection unit 315 being greater than or equal to a predetermined value. The ground may have coordinate axes on a horizontal plane, coordinates from pre-set map information may be used, or coordinates may be obtained from three-dimensional information of the ground detected by the imaging device 313.

[0046] The vehicle lift detection screen has a vehicle lift notification unit B. The vehicle lift notification unit B is the part that notifies the operator of the lift of the vehicle body of the work machine 10 (Figure 1). The vehicle lift notification unit B has a mode switching unit B1. The mode switching unit B1, like the mode switching unit A, is the part that allows the operator to select to switch from automatic driving mode to remote control mode. As shown in the example in Figure 4, in automatic driving mode, the mode switching unit B1 may be marked with something (for example, text) to indicate that it is switching to remote control mode. The heavy equipment operation mode in the display in Figure 4 corresponds to the remote control mode described above. For example, when the operator selects the mode switching unit B1, the control unit 43 (specifically the mode switching control unit 433) switches the user interface of the display unit 42 from automatic driving mode to remote control mode. Thus, the mode switching unit B1 is a switching reception area in the display image G for the input unit 41 to accept the switching between automatic driving mode and remote control mode. Thus, the control unit 43 detects and notifies the system when the posture of the work machine 10 is unstable in automatic driving mode. Furthermore, when the control unit 43 detects that the posture of the work machine 10 is unstable in automatic driving mode, it presents the mode switching unit B1 as a switching acceptance area. Note that the unstable posture is not limited to the vehicle body floating.

[0047] (Remote Control Mode) The display image G in remote control mode will be described below. Remote control mode is presented when switched from automatic driving mode. Remote control mode is a mode in which the input unit 41 and the display unit 42 present an interface for directly operating the work machine 10 (Figure 1). In remote control mode, three operation screens are displayed in sequence: a normal operation screen, a driving operation screen, and an attachment operation screen. In remote control mode, the work machine 10 to be operated may be automatically selected from the work machine 10 selected in automatic driving mode, or it may be selected by the operator when switching to remote control mode. When switching to remote control mode by the mode switching unit B1, it is preferable that the work machine 10 to be operated is automatically selected from the work machine 10 selected in automatic driving mode. In remote control mode, the control unit 43 (specifically, the remote control control unit 432) operates the work machine 10 in response to the operator's operation received by the input unit 41.

[0048] Incidentally, the working status of the working machine 10 may be presented at the timing of switching from the automatic driving mode to the remote operation mode. The working status may be presented only when the working machine 10 is switched to the remote operation mode after a sudden stop. The working status of the working machine 10 indicates the progress status of the automatic driving by the working machine 10 up to the above timing. For example, the type of work being executed by the working machine 10 (specifically, loading work, etc.) may be presented as the working status. Also, for example, the number of completed consecutive work phases (cycle number) may be presented as the working status. Also, for example, the number of times a predetermined phase has been executed may be presented as the working status. More specifically, the number of times the release phase has been executed (for example, the number of times earth and sand has been loaded onto the loading platform of the transport vehicle, etc.) may be presented as the working status. Also, for example, the amount of work completed may be presented as the working status. More specifically, the total amount of the work object released within the target release range (for example, the total amount of earth and sand loaded onto the loading platform of the transport vehicle, etc.) may be presented as the working status. The working status may be presented at the timing when switching to the remote operation mode is possible. For example, the working status may be presented on the vehicle body floating notification unit B (FIG. 4) of the vehicle body floating detection screen.

[0049] FIG. 5 is a diagram showing an example of a normal operation screen in the remote operation mode. The normal operation screen has a left lever operation unit C1L, a right lever operation unit C1R, a travel operation unit C1C, a mode switching unit C11, and an operation screen switching unit C12. The left lever operation unit C1L is arranged in the left part of the normal operation screen. The right lever operation unit C1R is arranged in the right part of the normal operation screen. The travel operation unit C1C is arranged between the left lever operation unit C1L and the right lever operation unit C1R. The left lever operation unit C1L and the right lever operation unit C1R are parts for allowing the operator to select the operation of the working machine 10 (FIG. 1). Specifically, what is actuated is the turning of the upper swing body 13 (FIG. 1) of the working machine 10, the operation of the boom 15a (FIG. 1), the operation of the arm 15b (FIG. 1), and the tip attachment 15c (FIG. 1).

[0050] For example, as shown in Figure 5, the left lever operating section C1L has an upper left lever C1L1, a lower left lever C1L2, a left left lever C1L3, and a right left lever C1L4. Specifically, the upper left lever C1L1 is the part that allows the operator to select an operation to rotate the arm 15b (Figure 1) in a direction that brings it closer to the boom 15a (Figure 1). The lower left lever C1L2 is the part that allows the operator to select an operation to rotate the arm 15b in a direction that moves it away from the boom 15a. The left left lever C1L3 is the part that allows the operator to select an operation to rotate the upper slewing body 13 (Figure 1) to the left when viewed from the operator's cab 13a (Figure 1) towards the boom 15a. The right left lever C1L4 is the part that allows the operator to select an operation to rotate the upper slewing body 13 to the right when viewed from the operator's cab 13a towards the boom 15a.

[0051] Furthermore, as shown in Figure 5, for example, the right lever operating section C1R has an upper right lever C1R1, a lower right lever C1R2, a left right lever C1R3, and a right right lever C1R4. Specifically, the upper right lever C1R1 is the part that allows the operator to select an operation to rotate the boom 15a (Figure 1) in a direction that brings it closer to the upper slewing body 13 (Figure 1). For example, the lower right lever C1R2 is the part that allows the operator to select an operation to rotate the boom 15a in a direction that moves it away from the upper slewing body 13. For example, the left right lever C1R3 is the part that allows the operator to select an operation to rotate the tip attachment 15c (Figure 1) in a direction that brings it closer to the arm 15b (Figure 1). For example, the right right lever C1R4 is the part that allows the operator to select an operation to rotate the tip attachment 15c in a direction that moves it away from the arm 15b. The correspondence between the up, down, left, and right parts of the left lever operating section C1L and the right lever operating section C1R and their respective operations may be changed in various ways.

[0052] Furthermore, the travel control unit C1C is the part that allows the operator to select whether to drive the left travel body 11bL (Figure 1) or the right travel body 11bR (Figure 1). The travel control unit C1C is the part that allows the operator to select whether to output a composite operation signal in a single operation, which includes a signal for controlling the drive of the left travel body 11bL and a signal for controlling the drive of the right travel body 11bR. For example, as shown in Figure 5, the travel control unit C1C has a travel control upper part C1C1, a travel control lower part C1C2, a travel control left part C1C3, a travel control right part C1C4, a travel control upper left part C1C5, a travel control upper right part C1C6, a travel control lower left part C1C7, and a travel control lower right part C1C8.

[0053] Figure 6 is a command signal table showing the command signals output when the travel control unit C1C is selected. For example, as shown in Figure 6, each of the control units C1C1 to C1C8 of the travel control unit C1C is assigned a command signal for controlling the driving of the left travel body 11bL (Figure 1) and the right travel body 11bR (Figure 1). That is, when an operator selects each of the control units C1C1 to C1C8, the command signals shown in the example in Figure 6 are transmitted to the travel motors 26 (Figure 1) that control the left travel body 11bL and the right travel body 11bR, respectively.

[0054] For example, as shown in Figure 6, the upper travel control section C1C1 allows the operator to select between moving the left travel body 11bL (Figure 1) forward and moving the right travel body 11bR (Figure 1) forward. As a result, the lower travel body 11 (Figure 1) moves the work machine 10 (Figure 1) forward. The lower travel control section C1C2 allows the operator to select between moving the left travel body 11bL backward and moving the right travel body 11bR backward. As a result, the lower travel body 11 moves the work machine 10 backward. The left travel control section C1C3 allows the operator to select between moving the left travel body 11bL backward and moving the right travel body 11bR forward. As a result, the lower travel body 11 makes the work machine 10 spin turn to the left. The right travel control section C1C4 allows the operator to select between moving the left travel body 11bL backward and moving the right travel body 11bR forward. As a result, the lower travel unit 11 spins the work machine 10 to the right. Also, the upper left travel control C1C5 is a part that allows the operator to choose between stopping the left travel unit 11bL and moving the right travel unit 11bR forward. As a result, the lower travel unit 11 pivots the work machine 10 to the left around the left travel unit 11bL as the axis. Also, the upper right travel control C1C6 is a part that allows the operator to choose between moving the left travel unit 11bL forward and stopping the right travel unit 11bR. As a result, the lower travel unit 11 pivots the work machine 10 to the right around the right travel unit 11bR as the axis. Also, the lower left travel control C1C7 is a part that allows the operator to choose between moving the left travel unit 11bL backward and stopping the right travel unit 11bR. As a result, the lower travel unit 11 pivots the work machine 10 to the left around the right travel unit 11bR as the axis. Furthermore, the lower right C1C8 of the travel operation controls allows the operator to choose between stopping the left travel unit 11bL and reversing the right travel unit 11bR. As a result, the lower travel unit 11 pivots the work machine 10 to the right around the left travel unit 11bL as the axis. Thus, the work machine 10 has a left travel unit 11bL and a right travel unit 11bR, which are driven according to command signals. The control unit 43 then outputs command signals to the work machine 10 for the left travel unit 11bL and the right travel unit 11bR based on a single operation to the travel operation unit C1C received by the input unit 41. In this case, the control unit 43 transmits multiple command signals to the work machine 10 corresponding to multiple drive units that are pre-associated with the single operation.

[0055] The mode switching unit C11 is the part that allows the operator to select whether to switch from remote control mode to automatic operation mode. As shown in the example in Figure 5, the mode switching unit C11 may be marked with something (for example, text) to indicate that it is switching to automatic operation mode. For example, when the operator selects the mode switching unit C11, the control unit 43 (specifically the mode switching control unit 433) switches the user interface of the display unit 42 from remote control mode to automatic operation mode by ending the current work mode, which is remote control mode. In this way, the mode switching unit C11 is a switching reception area in the display image G for the input unit 41 to accept the switching between automatic operation mode and remote control mode.

[0056] The operation screen switching unit C12 is the part that allows the operator to select which operation screen to switch. For example, if the operation screen switching unit C12 is selected on the normal operation screen, the operation screen will switch to the driving operation screen.

[0057] Figure 7 shows an example of a screen for driving operations in remote control mode. The normal operation screen has a driving operation unit C1C, a mode switching unit C11, an operation screen switching unit C12, and a work machine top display unit C13. The driving operation unit C1C and the mode switching unit C11 are the same as those on the normal operation screen. For example, if the operation screen switching unit C12 is selected on the driving operation screen, the operation screen switches to the attachment operation screen.

[0058] The top display section C13 of the work machine is a part for displaying the rotation angle of the upper slewing body 13 (Figure 1) relative to the lower traveling body 11 (Figure 1) of the work machine 10 (Figure 1). For example, as shown in Figure 7, the top display section C13 of the work machine may be a diagram simulating the work machine 10. Specifically, the top display section C13 of the work machine has a lower traveling body section C131 that simulates the lower traveling body 11 and an upper slewing body section C132 that simulates the upper slewing body 13. The top display section C13 of the work machine simulates a top view of the work machine 10 and shows the rotation position of the upper slewing body 13 relative to the lower traveling body section C131. Thus, the top display section C13 of the work machine shows the rotation angle of the upper slewing body 13 relative to the lower traveling body 11. The rotation angle of the upper slewing body 13 relative to the lower traveling body 11 may also be shown numerically. Furthermore, the image representing the rotation angle may be a diagram other than a top view of the work machine 10. For example, the image representing the rotation angle may be one that shows the rotation angle as a percentage of data in a pie chart.

[0059] Furthermore, the top display unit C13 of the work machine may be a part that allows the operator to select the rotation of the upper rotating body 13 (Figure 1) of the work machine 10 (Figure 1). For example, the position in which the upper rotating body 13 is rotated may be determined by dragging the upper rotating body unit C132.

[0060] Figure 8 shows an example of an attachment operation screen in remote control mode. The attachment operation screen includes a left lever operation section C1L, a right lever operation section C1R, a mode switching section C11, an operation screen switching section C12, and a work machine side display section C14. The left lever operation section C1L, the right lever operation section C1R, and the mode switching section C11 are the same as those on the normal operation screen. For example, if the operation screen switching section C12 is selected on the attachment operation screen, the operation screen switches to the attachment operation screen.

[0061] The side display unit C14 of the work machine is a part for displaying the current posture of the work machine 10 (Figure 1). The side display unit C14 of the work machine is also a part that allows the operator to select the operation of the attachment 15 (Figure 1) of the work machine 10. This is a diagram simulating the side display unit C14 of the work machine and the work machine 10. The side display unit C14 of the work machine simulates a side view of the work machine 10. Specifically, the top display unit C13 of the work machine has a machine body part C141, an arm part C143, and a tip attachment part C144. The machine body part C141 simulates the machine body 10a. The boom part C142 simulates the boom 15a (Figure 1). The arm part C143 simulates the arm 15b (Figure 1). The tip attachment part C144 simulates the tip attachment 15c (Figure 1).

[0062] Referring to Figure 9, a specific example of selecting the operation of attachment 15 (Figure 1) is shown below. In the example in Figure 9, an example of operating the arm 15b (Figure 1) of attachment 15 will be described. First, the control unit 43 allows the operator to select the component of attachment 15 to be operated. For example, the components of attachment 15 that the operator is allowed to select are the boom section C142, the arm section C143, and the tip attachment section C144. If any of the boom section C142, the arm section C143, or the tip attachment section C144 is selected, the control unit 43 identifies it as the component to be operated. At this time, the control unit 43 may also use the condition that the selection is a specific operation to identify the component to be operated. For example, the control unit 43 may identify the component to be operated when any of the boom section C142, the arm section C143, or the tip attachment section C144 is pressed and held down. The specific operation is not limited to this, and may also be a drag operation, for example. The fact that remote operation of the work machine 10 (Figure 1) is only possible when the input unit 41 accepts a specific operation may also apply to the left lever operation unit C1L, the right lever operation unit C1R, the travel operation unit C1C, and the work machine top display unit C13.

[0063] Once the components of attachment 15 (Figure 1) are identified, the identified components may be highlighted. The highlighting of the identified components may be indicated by shape, pattern, color, letters, symbols, etc. For example, in the example in Figure 9, the color of the identified arm portion C143 area is changed to be different from the other components. The control unit 43 then allows the operator to select the destination of the component. For example, the destination of the component is selected by dragging the component identified by touch operation. For example, in the example in Figure 9, the arm portion C143 is displayed in rotation by drag operation. Specifically, the arm portion C143 is displayed in rotation relative to the boom portion C142 to which the arm portion C143 is attached. At this time, the tip attachment portion C144 is displayed in rotation together with the arm portion C143 while maintaining its positional relationship with the arm portion C143. For example, although not shown, if the boom portion C142 is displayed in rotation, the boom portion C142 will be displayed in rotation relative to the machine body portion C141 to which the boom portion C142 is attached. Then, the arm section C143 and the tip attachment section C144 are displayed rotating together with the boom section C142 while maintaining their relative position to the boom section C142. In this way, one of the components of the attachment 15 is identified in response to a touch operation received by the input section 41. Then, in response to a drag operation received by the input section 41 after the touch operation, the identified component is displayed rotating on the machine body section 141c or the component on the side to which the identified component is attached. The control unit 43 then remotely operates the work machine 10 (Figure 1) by sending a command signal to the work machine 10 to move the identified component to its destination.

[0064] To rephrase what was described above, the control unit 43 inputs command signals to the display unit 42 for displaying images corresponding to each of the multiple components of the attachment. Then, the control unit 43 identifies an image of one of the multiple components in response to a touch operation received by the display unit 42 as an input unit 41. Furthermore, in response to a drag operation received by the input unit 41 after the touch operation, the control unit 43 rotates the image of the identified component against an image of the machine body 141c supporting the identified component or an image of another component. Then, in response to the drag operation, the control unit 43 transmits a command signal to the work machine 10 for remotely controlling the actual component of the work machine 10 corresponding to the image of the identified component.

[0065] Figure 10 shows an example of a screen for attachment operation in remote control mode. As shown in Figure 10, for example, if the work machine 10 (Figure 1) is in an unstable position, the control unit 43 may display the stable position section C15 on the display unit 42. The stable position section C15 is the part that displays the desired position of the work machine 10. As shown in Figure 10, it is preferable for the control unit 43 to display the stable position section C15 and the work machine side display section C14 superimposed. Also, as shown in Figure 10, it is preferable for the control unit 43 to display the stable position section C15 and the work machine side display section C14 in different ways. For example, the stable position section C15, which is the desired position, may be displayed in a transparent manner, and the work machine side display section C14 may be displayed in an opaque manner. The control unit 43 determines the desired position of the work machine 10 based on the tilt of the work machine 10 detected by the position detection unit 315 and the position of each component of the attachment 15 (Figure 1).

[0066] Furthermore, if the working machine 10 (Figure 1) is in an unstable position, the control unit 43 displays on the display unit 42 the operations required for the input unit 41 to bring the working machine 10 into a desirable position. The operations required to bring the working machine 10 into a desirable position may be indicated by shapes, patterns, colors, characters, symbols, etc. For example, in the example in Figure 10, the upper part C1R1 of the right lever, which is the part that allows the operator to select the operation to rotate the boom 15a (Figure 1) in the direction that brings it closer to the upper slewing body 13 (Figure 1), is changed to a different color from the other parts. As a result, it is possible to prevent the working machine 10 from malfunctioning due to an operation that would cause the working machine 10 to become even more unstable. For this purpose, in addition to emphasizing the operation in the direction of returning to a stable position as described above, it is also possible to not accept operations in the direction that leads to an unstable position. Furthermore, input by drag operation may be required. Moreover, input including a long press of a button may be required.

[0067] Furthermore, the control unit 43 allows remote operation if the working machine 10 (Figure 1) is in an unstable position and the input unit 41 receives an operation necessary to bring the working machine 10 into a desirable position. For example, in the example of Figure 10, only operation on the upper part C1R1 of the right lever, which is shown as an operation necessary to bring the working machine 10 into a desirable position, is permitted.

[0068] Each of the above operations (display, selection, setting, etc.) may also be referred to as a "step" in the method and program. Specifically, for example, selection may be referred to as a selection step. More specifically, the remote operation of the work machine 10 (Figure 1) may be referred to as a "remote operation step."

[0069] The effects of the autonomous driving system 1 shown in Figure 1 are as follows:

[0070] The automated driving system 1 comprises a work machine 10 and an information processing device 40. The work machine 10 is capable of automated driving. The information processing device 40 is connected to the work machine 10 in a communicative manner. The information processing device 40 includes a display unit 42 (Figure 2), an input unit 41 (Figure 2), and a control unit 43 (Figure 2). The display unit 42 displays information related to the work machine 10. The input unit 41 receives operations from the operator. The control unit 43 controls the display image G to be displayed on the display unit 42 and the information related to the input of the input unit 41. The control unit 43 performs automated driving processing and remote operation processing. In the automated driving processing, information related to automated driving is displayed on the display unit 42. In the automated driving processing, the input unit 41 receives input related to automated driving. In the remote operation processing, the work machine 10 is remotely operated based on the operations received by the input unit 41.

[0071] With the above configuration, the operator can remotely control the work machine 10 using the input unit 41 (Figure 2) which receives input information related to autonomous driving. For example, if the work machine 10 becomes unstable during autonomous driving, it can be easily restored to working order.

[0072] In the automated driving system 1, the control unit 43 (Figure 2) switches between an automated driving mode that displays information for performing automated driving processing and a remote control mode that displays information for performing remote control processing.

[0073] With the above configuration, it is possible to switch between displaying the automated driving process and the remote control process.

[0074] As shown in Figure 1, in the automatic driving system 1, the display unit 42 is a display. The control unit 43 (Figure 2) displays the mode switching unit A on the display unit 42 as a switching reception area for the input unit 41 to accept switching between the automatic driving mode and the remote control mode.

[0075] With the above configuration, the operator can switch between displaying the automated driving process and the remote control process.

[0076] As shown in Figure 1, in the automated driving system 1, the work machine 10 comprises a left travel body 11bL and a right travel body 11bR. As shown in Figure 6, the left travel body 11bL and the right travel body 11bR are each driven according to command signals. The control unit 43 (Figure 2) outputs multiple command signals to the work machine 10 based on a single operation received by the input unit 41 (Figure 2).

[0077] With the above configuration, the operator can perform multiple operations with a single operation.

[0078] As shown in Figure 1, in the automated driving system 1, the work machine 10 comprises an upper slewing body 13 and a lower traveling body 11 that supports the upper slewing body 13 so as to be rotatable. As shown in Figure 7, the control unit 43 (Figure 2) displays the slewing angle of the upper slewing body 13 relative to the lower traveling body 11 on the display unit 42.

[0079] With the above configuration, the rotation angle of the upper rotating body 13 relative to the lower traveling body 11 can be displayed to the operator, reducing the possibility of misjudging the direction of travel.

[0080] As shown in Figure 8, in the automatic driving system 1, the control unit 43 (Figure 2) displays the current posture of the work machine 10 (Figure 1) on the work machine side display unit C14 on the display unit 42 (Figure 2).

[0081] With the above configuration, the operator can easily grasp the position of the work machine 10 (Figure 1) using the display unit 42 (Figure 2).

[0082] As shown in Figure 10, in the automatic driving system 1, if the working machine 10 (Figure 1) is in an unstable position, the control unit 43 (Figure 2) displays the stable position C15, which is the desired position of the working machine 10, on the display unit 42 (Figure 2).

[0083] With the above configuration, the operator can easily understand the current position and desired position of the work machine 10 using the display unit 42.

[0084] As shown in Figure 10, in the automatic driving system 1, the control unit 43 (Figure 2) displays the stable posture section C15, which is the desired posture, and the work machine side display section C14, which is the current posture of the work machine 10, superimposed on the display unit 42 (Figure 2).

[0085] With the above configuration, the operator can easily grasp the desired position of the work machine 10 (Figure 1) relative to its current position using the display unit 42.

[0086] As shown in Figure 10, in the automatic driving system 1, the control unit 43 (Figure 2) displays the stable posture section C15, which is the desired posture, and the work machine side display section C14, which is the current posture, in different ways on the display unit 42 (Figure 2).

[0087] With the above configuration, the operator can more easily grasp the desired position of the work machine 10 relative to its current position using the display unit 42.

[0088] As shown in Figure 10, in the automatic driving system 1, if the working machine 10 (Figure 1) is in an unstable position, the control unit 43 (Figure 2) displays on the display unit 42 (Figure 2) the operations required for the input unit 41 (Figure 2) to bring the working machine 10 into a stable position C15, which is the desired position.

[0089] In the above configuration, the operator can easily understand the operations required to move the work machine 10 from its current position to a desired position using the display unit 42.

[0090] As shown in Figure 10, in the automatic driving system 1, the control unit 43 (Figure 2) allows remote operation when the working machine 10 (Figure 1) is in an unstable position and the input unit 41 (Figure 2) receives the necessary operation to bring the working machine 10 into a stable position C15, which is a desirable position.

[0091] With the above configuration, the operator can input the necessary operations to transition the work machine 10 from its current position to a desired position via the display unit 42 (Figure 2).

[0092] In the automated driving system 1, the control unit 43 (Figure 2) allows remote control when the input unit 41 (Figure 2) receives a specific operation (for example, a long press operation, a drag operation).

[0093] The above configuration reduces the possibility of the system being remotely controlled due to operator error.

[0094] As shown in Figure 4, in the automatic driving system 1, the control unit 43 (Figure 2) detects and notifies that the posture of the work machine 10 (Figure 1) is unstable in the automatic driving mode.

[0095] With the above configuration, it is possible to notify the worker that it is necessary to return the work machine 10 to a stable position.

[0096] As shown in Figure 4, in the automatic driving system 1, if the control unit 43 (Figure 2) detects that the posture of the work machine 10 (Figure 1) is unstable in the automatic driving mode, it presents the mode switching unit B1 as a switching acceptance area.

[0097] With the above configuration, it is possible to guide the operator to a remote control mode that returns the work machine 10 to a stable position.

[0098] In the automated driving system 1, the operational performance of the work machine 10 (Figure 1) is limited when it is remotely controlled compared to when it is automated.

[0099] With the above configuration, the operator can make fine adjustments to the posture of the work machine 10 via remote control.

[0100] As shown in Figure 1, in the automated driving system 1, the work machine 10 comprises a machine body 10a and an attachment 15. The attachment 15 comprises a boom 15a, an arm 15b, and a tip attachment 15c as its components. The boom 15a is rotatably attached to the machine body 10a. The arm 15b is rotatably attached to the boom 15a. The tip attachment 15c is rotatably attached to the arm 15b. The display unit 42 is a display. The input unit 41 is a touch panel provided on the display. The control unit 43 (Figure 2) identifies one of the components in response to a touch operation received by the input unit 41. In response to a drag operation received by the input unit 41 after the touch operation, the control unit 43 rotates and displays the identified component on the machine body 10a or the component to which the identified component is attached. The control unit 43 remotely controls the identified component in response to the drag operation.

[0101] With the above configuration, the attachment 15 of the work machine 10 can be intuitively operated remotely.

[0102] (Modifications) The above embodiment may be modified in various ways. For example, the number of components of the above embodiment (such as each part of the display image G) (including modifications) may be changed, and some components may not be provided. For example, modifications of the above embodiment may be combined in various ways. For example, the arrangement of components may be changed. For example, the inclusion relationships of components may be changed in various ways. For example, something described as a subordinate component included in a higher-level component may not be included in this higher-level component, but may be included in other components. For example, something described as multiple different members or parts may be treated as a single member or part. For example, something described as a single member or part may be divided and provided as multiple different members or parts. For example, the order of transitions of the display image G may be changed. For example, elements described as components of different screens may be displayed on the same screen. For example, each component may have only a part of each feature (function, arrangement, shape, operation, etc.).

[0103] For example, in remote control mode, the left lever operating section C1L, the right lever operating section C1R, and the travel operating section C1C may be in other configurations. Specifically, modifications of the left lever operating section C1L, the right lever operating section C1R, and the travel operating section C1C will be described with reference to Figure 11.

[0104] For example, as shown in Figure 11, the normal operation screen (image) in remote control mode may have a left lever operation section C2L, a right lever operation section C2R, and a travel operation section C2C. The left lever operation section C2L simulates a left lever (not shown) mounted on the work machine 10 and is the part that allows the operator to select the operation of the left lever. The left lever operation section C2L has a lever section C2L1, a lever play section C2L2, and a lever operation range section C2L3. The lever section C2L1 is the part that allows the operator to select the operation of the left lever. When the selected state of the lever section C2L1 is released, it returns to the neutral position (center position). The lever operation range section C2L3 is the part that displays the operable range of the lever section C2L1. The lever play section C2L2 is part of the lever operation range section C2L3 and is the part that displays the range in which the work machine 10 is not operated. In other words, even if the operator operates the lever C2L1 within the range of the lever play C2L2, the work machine 10 will not be activated.

[0105] The right lever operating section C2R simulates the right lever (not shown) mounted on the work machine 10 and allows the operator to select which lever to operate. For example, as shown in Figure 11, the right lever operating section C2R has a lever section C2R1 similar to the left lever operating section C2L, a lever play section C2R2, and a lever operating range section C2R3.

[0106] The travel control unit C2C simulates the left and right travel pedals (not shown) mounted on the work machine 10 and is the part that allows the operator to select which travel pedal to operate. For example, as shown in Figure 11, the travel control unit C2C has a left pedal operation unit C2CL and a right pedal operation unit C2CR. The left pedal operation unit C2CL is the part that allows the operator to select which travel pedal to operate. The left pedal operation unit C2CL has a left pedal section C2CL1, a left pedal play section C2CL2, and a left pedal operation range section C2CL3. The left pedal section C2CL1 is the part that allows the operator to select which left pedal to operate. The left pedal operation range section C2CL3 is the part that displays the operable range of the left pedal section C2CL1. The left pedal play section C2CL2 is part of the left pedal operation range section C2CL3 and is the part that displays the range in which the work machine 10 is not operated. In other words, even if the operator operates the left pedal C2CL1 within the range of the left pedal play C2CL2, the work machine 10 will not be activated.

[0107] The right pedal operation unit C2CR is the part that allows the operator to select which right driving pedal to operate. For example, as shown in Figure 11, the right pedal operation unit C2CR has a right pedal section C2CR1 similar to the left pedal operation unit C2CL, a right pedal play section C2CR2, and a right pedal operating range section C2CR3.

[0108] An automated driving system according to a first aspect of the present invention includes an information processing device that is communicatively connected to a work machine capable of automated driving. The information processing device includes a display unit that displays information about the work machine, an input unit that receives operations from an operator, and a control unit that controls the display image to be displayed on the display unit and the information input to the input unit through the operations. The control unit is capable of performing an automated driving process that displays information about automated driving on the display unit, receives input of information about automated driving at the input unit, and transmits commands related to automated driving to the work machine, and a remote operation process that transmits commands related to remote operation to the work machine and remotely operates the work machine based on the remote operation operations received by the input unit.

[0109] In the second aspect of the present invention, the automatic driving system, in the first aspect, allows the control unit to switch between an automatic driving mode, in which the display unit displays information for performing the automatic driving process, and a remote operation mode, in which it displays information for performing the remote operation process.

[0110] An automatic driving system according to a third aspect of the present invention, in the second aspect, the display unit is a touch panel type display capable of functioning as the input unit, and the control unit inputs a command signal to the display unit for displaying a switching reception area for accepting switching between the automatic driving mode and the remote control mode as the input unit.

[0111] An automatic driving system according to the fourth aspect of the present invention, in the first to third aspects, the work machine comprises a plurality of drive units, each of the plurality of drive units is driven in accordance with a command signal, and the control unit transmits a plurality of command signals corresponding to the plurality of drive units that are pre-associated with a single operation to the work machine based on a single operation received by the input unit.

[0112] An automatic driving system according to a fifth aspect of the present invention, in the first to fourth aspects, the work machine has an upper slewing body and a lower traveling body that rotatably supports the upper slewing body, and the control unit inputs a command signal to the display unit for displaying the slewing angle of the upper slewing body relative to the lower traveling body.

[0113] In the sixth aspect of the present invention, the automatic driving system, in the first to fifth aspects, includes a control unit inputting a command signal to the display unit for displaying the current posture of the work machine.

[0114] In the seventh aspect of the present invention, in the first to sixth aspects, the control unit inputs a command signal to the display unit to display a desired posture for the work machine when the posture of the work machine is unstable.

[0115] In the eighth aspect of the present invention, the automatic driving system, in the seventh aspect, includes a control unit inputting a command signal to the display unit for superimposing the desired posture and the current posture of the work machine.

[0116] In the ninth aspect of the present invention, the automatic driving system, in the eighth aspect, includes a control unit that inputs a command signal to the display unit for displaying the desired posture and the current posture in different manners.

[0117] In the tenth aspect of the present invention, in the first to ninth aspects, the control unit, when the posture of the work machine is unstable, inputs a command signal to the display unit to display the operation to be performed on the input unit necessary to bring the work machine into a desirable posture.

[0118] In the eleventh aspect of the present invention, in the first to tenth aspects, the control unit allows the execution of the remote control process when the input unit receives an operation necessary for the work machine to assume a desirable posture while the work machine is in an unstable posture.

[0119] In the twelfth aspect of the present invention, the automatic driving system, in the first to eleventh aspects, allows the control unit to execute the remote control process when the input unit receives a specific operation.

[0120] In the thirteenth aspect of the present invention, the automatic driving system, in the second aspect, has a control unit that detects and notifies that the posture of the work machine is unstable in the automatic driving mode.

[0121] In the fourteenth aspect of the present invention, in the third aspect, the control unit inputs a command signal to the display unit for displaying the switching reception area when it detects that the posture of the work machine is unstable in the automatic driving mode.

[0122] An automated driving system according to the fifteenth aspect of the present invention further comprises a work machine capable of performing automated driving in the first to fourteenth aspects, wherein the work machine's operating performance when remotely controlled is limited compared to its operating performance when automated driving is performed.

[0123] An automated driving system according to the sixteenth aspect of the present invention further comprises a work machine capable of automated driving in the sixth aspect, the work machine having a machine body and an attachment, the attachment including a plurality of components: a boom rotatably attached to the machine body, an arm rotatably attached to the boom, and a tip attachment rotatably attached to the arm, the display unit being a touch panel display capable of functioning as the input unit, the control unit inputting command signals to the display unit for displaying images corresponding to the plurality of components of the attachment, identifying an image of one of the images of the plurality of components in response to a touch operation received by the display unit as the input unit, rotating and displaying the image of the identified component on an image of the machine body supporting the identified component or an image of another component in response to the drag operation, and transmitting a command signal to the work machine for remotely operating a component of the work machine corresponding to the image of the identified component in response to the drag operation.

Claims

1. An automated driving system comprising an information processing device that is communicatively connected to a work machine capable of automated driving, wherein the information processing device includes a display unit that displays information about the work machine, an input unit that receives operations from an operator, and a control unit that controls the display image to be displayed on the display unit and the information input to the input unit, wherein the control unit is capable of performing an automated driving process that displays information about automated driving on the display unit, receives input of information about automated driving from the input unit, and transmits commands about automated driving to the work machine, and a remote operation process that transmits commands about remote operation to the work machine and remotely operates the work machine based on the remote operation operations received by the input unit.

2. An automated driving system according to claim 1, wherein the control unit is capable of switching the display unit between an automated driving mode for displaying information for performing the automated driving process and a remote control mode for displaying information for performing the remote control process.

3. An automated driving system according to claim 2, wherein the display unit is a touch panel type display capable of functioning as the input unit, and the control unit inputs a command signal to the display unit for displaying a switching reception area for accepting switching between the automated driving mode and the remote control mode as the input unit.

4. An automated driving system according to claim 1, wherein the work machine comprises a plurality of drive units, each of the plurality of drive units drives in accordance with a command signal, and the control unit transmits a plurality of command signals to the work machine, based on a single operation received by the input unit, the plurality of drive units corresponding to the single operation that is pre-associated with the single operation.

5. An automated driving system according to claim 1, wherein the work machine comprises an upper slewing body and a lower traveling body that rotatably supports the upper slewing body, and the control unit inputs a command signal to the display unit for displaying the slewing angle of the upper slewing body relative to the lower traveling body.

6. An automated driving system according to claim 1, wherein the control unit inputs a command signal to the display unit for displaying the current posture of the work machine.

7. An automated driving system according to claim 1, wherein the control unit inputs a command signal to the display unit for displaying a desired posture of the work machine when the posture of the work machine is unstable.

8. An automated driving system according to claim 7, wherein the control unit inputs a command signal to the display unit for superimposing the desired posture and the current posture of the work machine.

9. An automated driving system according to claim 8, wherein the control unit inputs command signals to the display unit for displaying the desired posture and the current posture in different manners.

10. An automated driving system according to claim 1, wherein the control unit, when the posture of the work machine is unstable, inputs a command signal to the display unit for displaying the operation to be performed on the input unit necessary to bring the work machine into a desirable posture.

11. An automated driving system according to claim 1, wherein the control unit allows the execution of the remote control process when the input unit receives an operation necessary for the working machine to assume a desirable posture while the working machine is in an unstable posture.

12. An automated driving system according to claim 1, wherein the control unit permits the execution of the remote control process when the input unit receives a specific operation.

13. An automatic driving system according to claim 2, wherein the control unit detects and notifies that the posture of the work machine is unstable in the automatic driving mode.

14. An automatic driving system according to claim 3, wherein the control unit inputs a command signal to the display unit for displaying the switching reception area when it detects that the posture of the work machine is in an unstable posture in the automatic driving mode.

15. An automated driving system according to claim 1, further comprising a work machine capable of automated driving, wherein the work machine's operating performance when remote operation is performed is limited compared to its operating performance when automated driving is performed.

16. An automated driving system according to claim 6, further comprising a work machine capable of automated driving, wherein the work machine comprises a machine body and an attachment, the attachment comprising a plurality of components: a boom rotatably mounted on the machine body, an arm rotatably mounted on the boom, and a tip attachment rotatably mounted on the arm, the display unit being a touch panel display capable of functioning as the input unit, the control unit inputting command signals to the display unit for displaying images corresponding to the plurality of components of the attachment, identifying an image of one of the images of the plurality of components in response to a touch operation received by the display unit as the input unit, rotating and displaying the image of the identified component against an image of the machine body supporting the identified component or an image of another component in response to a drag operation received by the input unit after the touch operation, and transmitting a command signal to the work machine for remotely controlling a component of the work machine corresponding to the image of the identified component in response to the drag operation.

Images