Indication system, indication method, and non-volatile storage medium

Abstract

An attachment (30) is attachable to and detachable from a coupler (8). An external sensor (20) acquires sensor data related to the attachment (30). A controller (50) outputs a guidance image to a monitor (40) on the basis of the sensor data acquired by the external sensor (20). The guidance image includes an indication that enables recognition of a relative position between a grip portion (8b) of the coupler (8) and a first pin (31) of the attachment (30) which is attached to and detached from the grip portion (8b).

Description

Display system, display method, and non-volatile memory medium

[0001] The present disclosure relates to a display system, a display method, and a non-volatile memory medium.

[0002] Regarding the alignment between an attachment and a tool carrier, for example, it is disclosed in European Patent Application Publication No. 4265849 (Patent Document 1). In Patent Document 1, in order to connect the tool carrier of a loader to an attachment, a visual movement instruction for moving the loader relative to the attachment is provided to an operator.

[0003] European Patent Application Publication No. 4265849

[0004] It may be difficult to align with the tool carrier when attaching an attachment, and further improvement in the efficiency of attaching and detaching the attachment is desired.

[0005] An object of the present disclosure is to provide a display system, a display method, and a non-volatile memory medium capable of further improving the efficiency of attaching and detaching an attachment.

[0006] The display system of the present disclosure includes a sensor, a monitor, and a controller. The sensor acquires sensor data related to an attachment that can be detached from a coupler. The controller outputs a guidance image to the monitor based on the sensor data. The guidance image includes a display capable of recognizing the relative position between the gripping portion of the coupler and the pin of the attachment to be detached from the gripping portion.

[0007] The display method of the present disclosure is a display method for displaying an image on a monitor to assist in attaching an attachment to a coupler. The display method in the working machine of the present disclosure includes a step of acquiring sensor data related to an attachment and a step of outputting a guidance image to the monitor based on the acquired sensor data. The guidance image includes a display capable of recognizing the relative position between the gripping portion of the coupler and the pin of the attachment to be detached from the gripping portion.

[0008] The non-transitory storage medium of this disclosure stores a program that causes the controller's processor to perform the steps of acquiring sensor data related to an attachment and outputting a guidance image to a monitor, based on the acquired sensor data, which includes a display that allows recognition of the relative position between the gripping portion of the coupler and the pins of the attachment that are attached to and detached from the gripping portion.

[0009] According to this disclosure, a display system, a display method, and a non-volatile storage medium can be realized that enable further efficiency in attachment and detachment.

[0010] This is a perspective view showing the configuration of a work machine in one embodiment of the present disclosure. This is a diagram for explaining the operation of the coupler when an attachment is mounted on the work machine shown in Figure 1. This is a functional block diagram showing the configuration of the controller in the display system of the work machine shown in Figure 1. This is a flowchart showing the display method of the work machine shown in Figure 1. This is a diagram showing a guidance image displayed in three dimensions. This is a diagram showing a guidance image including an arrow (support arrow) indicating the direction of movement to the target position. This is a diagram showing a guidance image including an image of the target position and a coupler rotation instruction. This is a diagram showing a guidance image including lines and dimensions for operation support. This is a diagram showing a guidance image including an angle for operation support in a top view. This is a diagram showing a guidance image that supports the operation of aligning the coupler directly with the attachment in a top view.

[0011] The embodiments of this disclosure will be described below with reference to the drawings. In the specification and drawings, the same components or corresponding components are denoted by the same reference numerals, and redundant descriptions are avoided. In addition, in the drawings, some components may be omitted or simplified for the sake of clarity. Furthermore, at least some of the embodiments and modifications may be combined with each other as desired.

[0012] In the following explanation, "up," "down," "front," "rear," "left," and "right" refer to directions relative to the operator seated in the driver's seat 4S within the cab 4 shown in Figure 1. A side view refers to a viewpoint from which the work machine 100 is viewed from the left or right. A top view refers to a viewpoint from which the work machine 100 is viewed from above and below.

[0013] <Configuration of the Working Machine> As an example of the working machine of this disclosure, the configuration of a hydraulic excavator will be explained with reference to Figure 1. However, the working machine to which this disclosure applies is not limited to a hydraulic excavator, and may be other working machines such as a backhoe.

[0014] Figure 1 is a schematic diagram showing the configuration of a hydraulic excavator as an example of a work machine in one embodiment of the present disclosure. As shown in Figure 1, the hydraulic excavator 100 of this embodiment has a main body 1 and a work machine 2 that is operated by hydraulics. The main body 1 has a slewing body 3 and a traveling body 5.

[0015] The vehicle 5 has a pair of tracks 5Cr and a drive motor 5M. The hydraulic excavator 100 can move by the rotation of the tracks 5Cr. The drive motor 5M is provided as the drive source for the vehicle 5. The drive motor 5M is a hydraulic motor that is operated by hydraulic pressure. The vehicle 5 may also have wheels (tires).

[0016] The slewing body 3 is positioned on and supported by the traveling body 5. The slewing body 3 is capable of swiveling relative to the traveling body 5 about a pivot axis RX by a slewing motor (not shown). The pivot axis RX is the pivot center of the slewing body 3. The slewing motor is a hydraulic motor that operates by hydraulic pressure. The pivot axis RX is a hypothetical straight line that serves as the pivot center of the slewing body 3. Note that the traveling motor 5M or the slewing motor may be electric motors.

[0017] The slewing body 3 has a cab 4. Inside the cab 4 is a driver's seat 4S where the operator sits. The operator (crew) can sit in the cab 4 and perform operations such as attaching and detaching the attachment 30 to the coupler 8, operating the work implement 2, rotating the slewing body 3 relative to the vehicle 5, and driving the hydraulic excavator 100 using the vehicle 5. The hydraulic excavator 100 may be remotely controlled.

[0018] The implement 2 is attached to the slewing body 3. The implement 2 has a boom 6, an arm 7, a boom cylinder 10, an arm cylinder 11, and an attachment cylinder 12. The coupler 8 is attached to the implement 2 and is attached to the tip of the arm 7.

[0019] The boom 6 is rotatably connected to the main body 1. Specifically, the base end of the boom 6 is rotatably connected to the slewing body 3 with the boom foot pin 13 as the pivot point. The arm 7 is rotatably connected to the boom 6. Specifically, the base end of the arm 7 is rotatably connected to the tip of the boom 6 with the boom top pin 14 as the pivot point. The coupler 8 is rotatably connected to the arm 7. Specifically, one end of the coupler 8 is rotatably connected to the tip of the arm 7 with the arm top pin 15 as the pivot point.

[0020] One end of the boom cylinder 10 is connected to the slewing body 3, and the other end is connected to the boom 6. The boom 6 is movable relative to the main body 1 by the boom cylinder 10. By operating the boom cylinder 10, the boom 6 can rotate vertically relative to the slewing body 3 with the boom foot pin 13 as the pivot point.

[0021] One end of the arm cylinder 11 is connected to the boom 6, and the other end is connected to the arm 7. The arm 7 is movable relative to the boom 6 by the arm cylinder 11. The movement of the arm cylinder 11 allows the arm 7 to rotate in the digging direction or the dumping direction, with the boom top pin 14 as the pivot point.

[0022] One end of the attachment cylinder 12 is connected to the arm 7, and the other end is connected to the link 17. The link 17 is connected to the other end of the coupler 8 by a link pin 16. The coupler 8 is movable relative to the arm 7 by the attachment cylinder 12. Due to the movement of the attachment cylinder 12, the coupler 8 is rotatable around the arm top pin 15 as a pivot point.

[0023] Each of the boom cylinder 10, arm cylinder 11, and attachment cylinder 12 is a hydraulic cylinder driven by hydraulic pressure, but other actuators such as electric cylinders may also be used.

[0024] The coupler 8 is configured to allow attachment 30 to be attached and detached. The coupler 8 makes it easy to replace the attachment 30 that is attached to the tip of the work implement 2. The attachment 30 that can be attached to the tip of the work implement 2 may be a bucket, breaker, grapple, large crusher, small crusher, etc. In Figure 1, a bucket is shown as an example of an attachment 30.

[0025] The coupler 8 includes, for example, a coupler body 8a and a locking pin 8c. The coupler body 8a has holes for supporting the arm top pin 15 and the link pin 16, respectively. The coupler body 8a also has a gripping portion 8b. The gripping portion 8b is, for example, a recess and has a hook shape. The first pin 31 of the attachment 30 can be inserted into the gripping portion 8b. By inserting the first pin 31 into the gripping portion 8b, the first pin 31 can be hooked onto the coupler 8.

[0026] The locking pin 8c is attached to the coupler body 8a. For example, two locking pins 8c are attached to the coupler body 8a. The locking pin 8c is controlled to move between, for example, a retracted state and an extended state within the coupler body 8a. When the locking pin 8c is extended from the coupler body 8a (locked state), the coupler body 8a and the locking pin 8c form a pin locking portion. The second pin 32 of the attachment 30 can be inserted into this pin locking portion. When the second pin 32 is inserted into the pin locking portion, the second pin 32 is sandwiched between the coupler body 8a and the locking pin 8c and locked into the pin locking portion. When the locking pin 8c is retracted into the coupler body 8a (unlocked state), the locking state of the second pin 32 is released.

[0027] An external sensor 20 is attached to the hydraulic excavator 100. The external sensor 20 is attached to, for example, the operator's cab 4 of the hydraulic excavator 100. The external sensor 20 may be attached to the top of the operator's cab 4, or to the side of the operator's cab 4. The external sensor 20 may be attached to parts of the hydraulic excavator 100 other than the operator's cab 4, or it may be installed at a location away from the hydraulic excavator 100.

[0028] The external sensor 20 acquires sensor data related to the attachment 30. The external sensor 20 is, for example, an imaging device. The external sensor 20 may be, for example, a monocular camera that acquires a two-dimensional image of the attachment 30. Alternatively, the external sensor 20 may be, for example, a depth camera. As a depth camera, for example, a stereo camera, a structured light camera, or a time-of-flight camera may be used. The external sensor 20 is not limited to an imaging device as long as it can acquire sensor data related to the attachment 30, and may also be, for example, a LiDAR (Light Detection and Ranging). Thus, the external sensor 20 includes cameras, LiDAR, etc. Therefore, the sensor data related to the attachment 30 includes images of the attachment 30 captured by the camera, point cloud data of the attachment 30 acquired by the LiDAR, etc. Furthermore, the sensor that acquires sensor data related to the attachment 30 is not limited to the external sensor, but can be any sensor that can acquire sensor data related to the attachment 30.

[0029] The hydraulic excavator 100 has an attitude sensor 21 (Figure 3) and a position and orientation sensor 24 (Figure 3). The attitude sensor 21 detects the attitude of the work implement 2 and outputs an attitude signal indicating the attitude of the work implement 2. The attitude sensor 21 can detect the attitude of the boom 6, arm 7, and coupler 8. The attitude sensor 21 includes sensors placed on each of the boom 6, arm 7, and coupler 8. The attitude sensor 21 may be, for example, an IMU (Inertial Measurement Unit), a stroke sensor, a potentiometer, an imaging device, etc.

[0030] The position and orientation sensor 24 (Figure 3) includes, for example, a GNSS (Global Navigation Satellite Systems) receiver and a GNSS antenna. The position and orientation sensor 24 includes two GNSS antennas 23a and 23b (Figures 1 and 3). Each of the two GNSS antennas 23a and 23b is installed at a different position on the rotating body 3. The GNSS receiver (not shown) calculates the position of the rotating body 3 in the global coordinate system and the direction in which the rotating body 3 is facing from the satellite positioning signals received from the satellite by each of the GNSS antennas 23a and 23b. The GNSS receiver outputs a position signal indicating the position of the rotating body 3 and an orientation signal indicating the direction in which the rotating body 3 is facing.

[0031] The position and orientation sensor 24 has a rotation angle sensor 22 (Figure 3). The rotation angle sensor 22 is fixed to the rotating body 3, for example. The rotation angle sensor 22 detects the rotation angle of the rotating body 3 relative to the traveling body 5 and outputs a rotation angle signal indicating the rotation angle of the rotating body 3. The rotation angle sensor 22 can detect the rotation angle in the machine coordinate system (local coordinate system). The rotation angle sensor 22 may be, for example, an IMU, a potentiometer, an imaging device, etc. The machine coordinate system is a Cartesian coordinate system with the rotation center of the rotating body 3 as the origin, and represented by an axis extending in the front-rear direction, an axis extending in the left-right direction, and an axis extending in the up-down direction (rotation axis RX: Figure 1).

[0032] <Attachment Installation Operation> Next, the operation of attaching the attachment 30 to the work machine shown in Figure 1 will be explained using Figure 2.

[0033] Figure 2 is a diagram illustrating the operation of the coupler when an attachment is mounted on the work machine shown in Figure 1. As shown in Figure 2(A), the hydraulic excavator 100 moves to a position where the attachment 30 can be mounted. The position where the hydraulic excavator 100 can mount the attachment 30 is the position where the gripping portion 8b of the coupler 8 reaches the first pin 31 of the attachment 30 due to the operation of the work machine 2, and where the coupler 8 is directly facing the attachment 30. The position where the coupler 8 is directly facing the attachment 30 will be described later using Figure 10.

[0034] In this state, the lock pin 8c of the coupler 8 is in a locked position, for example, protruding from the coupler body 8a. The circle 60 shown by the dashed line in the figure is the target position of the arm top pin 15 when the attachment 30 is being mounted.

[0035] As shown in Figure 2(B), the arm 7 and boom 6 are driven, causing the arm top pin 15 to move toward the target position 60, thereby moving the coupler 8 closer to the attachment 30. As the coupler 8 moves toward the target position 60, the lock pin 8c is retracted into the coupler body 8a, entering an unlocked state.

[0036] As shown in Figure 2(C), the above movement of the coupler 8 causes the arm top pin 15 to reach the target position 60. In this state, the first pin 31 of the attachment 30 is inserted into the gripping portion 8b of the coupler 8.

[0037] As shown in Figure 2(D), the coupler 8 rotates relative to the arm 7 while the first pin 31 remains inserted in the gripping portion 8b. This moves the coupler 8 to the mounting position of the attachment 30. At this mounting position, the mounting surface (bottom surface) 8d of the coupler 8 is aligned with a part of the surface of the attachment 30. Up to this point, the attachment 30 remains stationary on the ground without moving.

[0038] As shown in Figure 2(E), the coupler 8 moves to lift the attachment 30 off the ground. At this time, the gripping portion 8b of the coupler 8 holds the first pin 31 of the attachment 30, and the mounting surface 8d of the coupler 8 supports a portion of the surface of the attachment 30.

[0039] As shown in Figure 2(F), the locking pin 8c is positioned in a locked state, protruding from the coupler body 8a. This causes the second pin 32 of the attachment 30 to be sandwiched and locked between the locking pin 8c and the coupler body 8a. As a result, the attachment 30 is locked to the coupler 8 and attached to the coupler 8.

[0040] In this embodiment, the attachment operation of the attachment 30 is performed as described above. The attachment operation of the attachment 30 is performed by the operation of an operator who has boarded the cab 4 or by the remote operation of an operator at a remote location away from the hydraulic excavator 100.

[0041] In the above description, the attachment method of the attachment 30 of the type that scoops up the attachment 30 with the coupler 8 and then locks it has been described. However, the present disclosure is also applicable to an attachment 30 of the type that locks the attachment 30 to the coupler 8 without scooping it up.

[0042] <Display System and Display Method> Next, the display system and display method in this embodiment will be described.

[0043] FIG. 3 is a functional block diagram showing the configuration of a controller in the display system of the work machine shown in FIG. 1. FIG. 4 is a flowchart showing the display method in the work machine shown in FIG. 1.

[0044] In the following description, as an example of sensor data related to the attachment 30, an image of the attachment 30 (an image captured by a camera such as a monocular camera or a depth camera) will be described. However, the image of the attachment 30 in the following description may be information equivalent to an image (for example, point cloud data by LiDAR), as long as it is sensor data related to the attachment 30.

[0045] As shown in FIG. 3, the controller 50 acquires an image of the attachment 30 acquired by the external sensor 20. The controller 50 outputs a guidance image including the relative position between the hydraulic excavator 100 and the attachment 30 to the monitor 40 based on the image of the attachment 30 detected by the external sensor 20. The guidance image includes a display in which the relative position between the gripping portion 8b of the coupler 8 and the first pin 31 of the attachment 30 that is detached from the gripping portion 8b can be recognized.

[0046] The controller 50 acquires information regarding the attitude of the work machine 2 detected by the attitude sensor 21. The controller 50 acquires information regarding the position and orientation of the slewing body 3 calculated by the GNSS receiver based on satellite positioning signals received by the GNSS antennas 23a and 23b. The controller 50 also acquires information regarding the slewing angle of the slewing body 3 relative to the traveling body 5 detected by the slewing angle sensor 22. The monitor 40 is composed of a graphic display such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The monitor 40 may be attached to the hydraulic excavator 100, or it may be attached to a remote controller for remotely operating the hydraulic excavator 100.

[0047] As shown in Figures 3 and 4, the controller 50 includes an acquisition unit 50a, a selection unit 50b, an image generation unit 50c, an output control unit 50d, and a memory 50e. The acquisition unit 50a acquires an image of the attachment 30 detected by the external sensor 20 (Step S1: Figure 4). The controller 50 detects the attachment 30 based on the acquired image of the attachment 30 (Step S2: Figure 4). The detection of the attachment 30 by the controller 50 may also be performed by an operator touching the image area of ​​the attachment 30 on the monitor 40.

[0048] The selection unit 50b selects a candidate attachment based on the image of the attachment 30 acquired from the acquisition unit 50a (step S3: FIG. 4). When selecting a candidate attachment, the selection unit 50b refers to a list prestored in the memory 50e. The list stored in the memory 50e includes information related to the shapes of a plurality of candidate attachments. The memory 50e may store, as information related to the shape of a candidate attachment, for example, 3D CAD (Computer-Aided Design) data of the candidate attachment. The selection unit 50b selects a candidate attachment that matches the image of the attachment 30 from among the plurality of candidate attachments included in the list. Specifically, the selection unit 50b selects a candidate attachment that matches the image of the attachment 30 based on a comparison between the image of the attachment 30 acquired from the acquisition unit 50a and the information related to the shape of the candidate attachment. When the selection unit 50b determines that there is an attachment identical to the attachment 30 acquired from the acquisition unit 50a among the plurality of candidate attachments included in the list, it uses the information related to the shape of the identical candidate attachment. At this time, an error may be displayed on the monitor 40 when the image of the attachment 30 and the information related to the shape of the candidate attachment do not match. When they do not match, the selection unit 50b may select a candidate attachment with the highest degree of shape similarity. The controller 50 may display a plurality of candidate attachments with a high degree of shape similarity and have the operator make a selection.

[0049] In some cases, the attachment 30 may be replaced with one not included in the above list. In this case, the attachment 30 is photographed by an imaging device mounted on the hydraulic excavator 100 while the excavator 100 is being moved. This allows the three-dimensional shape of the attachment 30 to be acquired, and 3D CAD data may be created and stored in the list. Alternatively, the attachment 30 may be photographed by the operator using a handheld camera (for example, a camera mounted on a mobile information terminal) while changing the viewpoint, thereby creating 3D CAD data of the attachment 30 and storing it in the list. Furthermore, the operator may physically set the position of the first pin 31.

[0050] Memory 50e stores dimensional data related to the hydraulic excavator 100. The dimensional data may include, for example, the distance between the GNSS antennas 23a and 23b and the boom foot pin 13, the distance between the boom foot pin 13 and the boom top pin 14, the distance between the boom top pin 14 and the arm top pin 15, and data related to the shapes of the boom 6, arm 7, and coupler 8 on the hydraulic excavator 100. The dimensional data stored in memory 50e may also include the position of the external sensor 20 on the hydraulic excavator 100 and the installation angle of the external sensor 20.

[0051] The controller 50 acquires information such as the type, shape, and dimensions of the attachment 30 based on information related to the shape of the selected candidate attachment, and estimates the relative position and orientation of the attachment 30 with respect to the hydraulic excavator 100. The controller 50 also estimates the position and orientation of at least the coupler 8 on the hydraulic excavator 100 based on information acquired from the attitude sensor 21, information acquired from the position and orientation sensor 24, and dimensional data stored in the memory 50e. Furthermore, the controller 50 can improve the positional accuracy of the attachment 30 with respect to the hydraulic excavator 100 by referring to the position of the external sensor 20 on the hydraulic excavator 100.

[0052] The image generation unit 50c acquires information related to the shape of the candidate attachment selected by the selection unit 50b and generates a guidance image based on the selected candidate attachment (Step S4: Figure 4). Specifically, the image generation unit 50c generates a guidance image that visualizes the position and orientation of the attachment 30 and the coupler 8, respectively, as estimated by the controller 50. The guidance image includes an image showing the relative position between the attachment 30 and the hydraulic excavator 100. The relative position may also include the relative angle between the gripping part 8b and the first pin 31c. Specific examples of guidance images will be described later.

[0053] The output control unit 50d outputs the guidance image acquired from the image generation unit 50c to the monitor 40 (Step S5: Figure 4). As a result, the guidance image is displayed on the monitor 40 (Step S6: Figure 4). The guidance image is an image that allows recognition of the relative position between the gripping portion 8b of the coupler 8 and the first pin 31c of the attachment 30 that is attached to and detached from the gripping portion 8b, as described above.

[0054] The controller 50 includes a processor, main memory, and storage. The processor is, for example, a CPU (Central Processing Unit). The main memory includes, for example, non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory).

[0055] The controller 50 reads the program stored in storage, loads it into main memory, and executes predetermined processing according to the program. The program may also be distributed to the controller 50 via a network.

[0056] The controller 50 may be mounted on the hydraulic excavator 100, or it may be located separately outside the hydraulic excavator 100. If the controller 50 is located separately outside the hydraulic excavator 100, the controller 50 may be wirelessly connected to the external sensor 20, attitude sensor 21, slewing angle sensor 22, etc. The controller 50 may be stored in a server located away from the hydraulic excavator 100.

[0057] <Examples of Guidance Images> Next, examples of guidance images displayed on the monitor 40 will be explained using Figures 2, 5 to 12.

[0058] The guidance image may be a two-dimensional image representing the coupler 8 and attachment 30 in a side view, as shown in Figure 2. Alternatively, the guidance image may be a three-dimensional image representing the coupler 8 and attachment 30 in an oblique view, including a side view, as shown in Figure 5.

[0059] As shown in Figure 5, the guidance image only needs to display at least the gripping portion 8b and the first pin 31 of the coupler 8. However, the guidance image may also display the coupler 8, attachment 30, arm 7, link 17, etc., and may also display the entire hydraulic excavator 100. Furthermore, the components displayed in the guidance image may be displayed as semi-transparent skeleton images with the background transparent. For example, the part of the attachment 30 other than the first pin 31 may be displayed as a skeleton image. Similarly, the parts of the hydraulic excavator 100 other than the coupler 8 may be displayed as skeleton images.

[0060] Furthermore, the guidance images may include symbols such as arrows to assist the operator in their actions, as shown in Figures 5 and 6. These operation-assisting arrows, for example, indicate the direction in which to move the coupler 8 to the target position when attaching or detaching an attachment.

[0061] The above target position may be the position of the coupler 8 in the inserted state, where the first pin 31 is inserted into the gripping portion 8b of the coupler 8, or it may be an intermediate position in the transition from the current position to the inserted state. In Figure 6, a circuit breaker is shown as an example of the attachment 30.

[0062] The operator can recognize symbols such as arrows in the guidance image to determine the next direction of movement for the coupler 8, thereby enabling more efficient attachment and detachment of the attachment 30.

[0063] Furthermore, as shown in Figure 7, the guidance image may also show the arm 7 and coupler 8 at the target position in addition to the arm 7 and coupler 8 at the current position. In Figure 7, the arm 7 and coupler 8 at the current position are shown with solid lines, and the arm 7 and coupler 8 at the target position are shown with dashed lines.

[0064] In the guidance image, it is preferable that the arm 7 and coupler 8 at the target position are displayed in a different format from the arm 7 and coupler 8 at the current position so that they can be easily distinguished from the arm 7 and coupler 8 at the current position. The arm 7 and coupler 8 at the target position may be displayed, for example, as a skeleton image with a transparent background.

[0065] The target position described above may be the inserted state in which the first pin 31 is inserted into the gripping portion 8b of the coupler 8, or it may be an intermediate state in the process of transitioning from the current position to the inserted state. Note that in Figure 7, the attachment 30 is shown in a simplified manner. In Figure 7, the arm 7 and coupler 8 at the current position and the arm 7 and coupler 8 at the target position are shown, but either the arm 7 or the coupler 8 may be shown separately.

[0066] By recognizing the coupler 8 at the target position using the guidance image, the operator can understand the next transition state of the coupler 8, making it possible to perform the attachment and detachment operation of the attachment 30 more efficiently.

[0067] The guidance image may also include at least one of the lines and dimensions for operation assistance, as shown in Figure 8. The lines for operation assistance include, for example, a plurality of straight lines L1a, L1b, L2a, L2b, L3a, L3b.

[0068] The lines L1a and L1b are, for example, a horizontal and a vertical line passing through the center of the arm top pin 15 in a side view of the hydraulic excavator 100. The lines L2a and L2b are, for example, a horizontal and a vertical line passing through the center of the gripping section 8b in a side view of the hydraulic excavator 100. The center of the gripping section 8b is the center of the inscribed or circumscribed circle of the gripping section 8b in a side view. The lines L3a and L3b are, for example, a horizontal and a vertical line passing through the center of the first pin 31 in a side view of the hydraulic excavator 100.

[0069] Lines L1a, L2a, and L3a are all parallel to each other. Lines L1b, L2b, and L3b are all parallel to each other. Lines L1a and L1b are perpendicular to each other, lines L2a and L2b are perpendicular to each other, and lines L3a and L3b are perpendicular to each other.

[0070] The lines used for operational support can be any lines that the operator uses as a reference point when attaching or detaching the attachment 30, and are not limited to the horizontal or vertical lines mentioned above.

[0071] By recognizing the lines used for operation assistance in the guidance image, the operator can determine the positional misalignment of the arm top pin 15, gripping portion 8b, first pin 31, etc., thereby enabling more efficient attachment and detachment of the attachment 30.

[0072] The dimensions for operation assistance include multiple dimensions X and Y. Dimension X is, for example, the horizontal distance between the center of the arm top pin 15 and the center of the first pin 31 in a side view of the hydraulic excavator 100. Dimension Y is, for example, the vertical distance between the center of the gripping portion 8b and the center of the first pin 31 in a side view of the hydraulic excavator 100.

[0073] By recognizing the dimensions for operational assistance in the guidance image, the operator can accurately determine the amount of misalignment between the arm top pin 15, gripping portion 8b, first pin 31, etc., thereby enabling more efficient attachment and detachment of the attachment 30.

[0074] The guidance image includes the relative position of the gripping portion 8b and the first pin 31c, as described above. The relative position of the gripping portion 8b and the first pin 31c includes the relative angle between the gripping portion 8b and the first pin 31c. The relative angle is, as shown in Figure 9, for example, the angle θ1 between a straight line L2 extending along the axial direction of the first pin 31 in a top view and a straight line L1 extending along the axial direction of the gripping portion 8b. The axial direction of the gripping portion 8b refers to the axial direction of the first pin 31 when the first pin 31 is inserted into the gripping portion 8b. The relative angle may also be the angle θ2 between a straight line L4 perpendicular to the first pin 31 and a straight line L3 extending from the base end to the tip end of the boom 6. In this guidance image, the entire arm 7 or its tip and the coupler 8 may be hidden, or they may be shown as a skeleton.

[0075] By recognizing angles θ1 and θ2 in the guidance image, the operator can determine the difference in angle between the gripping portion 8b and the first pin 31 in a top view, thereby enabling more efficient attachment and detachment of the attachment 30.

[0076] The guidance image includes instructions for operating the slewing body 3 and the traveling body 5 so that the relative angle between the gripping portion 8b and the first pin 31 in a top view is reduced, as shown in Figure 10. These instructions include instructions for traveling the traveling body 5 (white arrows in Figure 10) and instructions for rotating the slewing body 3 (solid black arrows in Figure 10) so that the hydraulic excavator 100 is in the state shown by the dashed line in Figure 10. Specifically, the guidance image displays instructions for traveling the traveling body 5 and rotating the slewing body 3 so that in a top view, the straight line L3 extending from the base end to the tip end of the boom 6 coincides with the straight line L4 perpendicular to the first pin 31 (so that the angle θ2 becomes zero). The guidance image may also include instructions for traveling the traveling body 5 in a direction parallel to the straight line L2 extending in the axial direction of the first pin 31. The guidance image may also include instructions for traveling the traveling body 5 in a direction different from the straight line L2 (a direction that is not parallel).

[0077] In the top view, the state in which line L3 coincides with line L4 is the state in which the coupler 8 is directly facing the attachment 30. The state of the hydraulic excavator 100, shown by the dashed line in Figure 10, may also be shown as a skeleton.

[0078] The guidance image includes an indication that the traveling body 5 should be operated so that its extension direction L5 and the axial direction L2 of the first pin 31 are perpendicular or parallel to each other, as an instruction to align the coupler 8 directly with the attachment 30. The indication is, for example, the dashed arrow and the dashed line L5A in Figure 10. The dashed arrow indicates the direction of change of the extension direction L5. The dashed line L5A is a line that extends parallel to the axial direction L2 of the first pin 31 in a top view, and indicates the target extension direction of the traveling body 5. For the sake of explanation, the indication of the direction of change is shown as a dashed arrow, but it is not limited to this, and any indication that shows the direction of change is acceptable. Similarly, the indication of the target extension direction is not limited to the dashed line 5LA, but any indication that shows the target extension direction of the traveling body 5 is acceptable.

[0079] Upon seeing the dashed arrow on the monitor 40, the operator changes the extension direction L5 of the vehicle 5 to extension direction L5A, for example, by operating only one of the pair of tracks 5Cr. In the above description, an indicator showing an instruction to make the extension direction L5 of the vehicle 5 parallel to the axial direction L2 of the first pin 31 was explained, but an indicator showing an instruction to make the extension direction L5 of the vehicle 5 perpendicular to the axial direction L2 of the first pin 31 may also be used.

[0080] By recognizing the above operation instructions in the guidance image, the operator can understand the operation of the traveling body 5 and the rotating body 3 to align the coupler 8 directly with the attachment 30, thereby enabling more efficient attachment and detachment of the attachment 30.

[0081] Furthermore, the remote controller for remote operation may display the captured image from the camera and the guidance image simultaneously, or they may be switched between.

[0082] <Effects> Next, the effects of this embodiment will be described.

[0083] When attaching or detaching the attachment 30 shown in Figure 2, it is necessary to move the work machine 2 and the slewing body 3, but it is difficult to grasp the positional relationship of the attachment 30 in the depth direction. In addition, the gripping portion 8b of the coupler 8 may be hidden from view by the attachment 30. For this reason, it is difficult to align the gripping portion 8b of the coupler 8 with the first pin 31 of the attachment 30 when attaching or detaching the attachment 30. In particular, when attaching or detaching by remote control, the operator must perform the operation using only camera information, making the operation even more difficult. For this reason, there was a request for a way to efficiently attach the attachment 30.

[0084] In this embodiment, as shown in Figure 5 and other figures, the guidance image displayed on the monitor 40 includes a display that allows the operator to recognize the relative position between the gripping portion 8b of the coupler 8 and the first pin 31 of the attachment 30. This allows the operator to attach and detach the attachment 30 to the coupler 8 while confirming the relative position between the gripping portion 8b and the first pin 31 using the guidance image on the monitor 40. Therefore, the attachment and detachment of the attachment 30 can be made more efficient not only when the operator is on board the hydraulic excavator 100 to perform the operation, but also when the operator performs the operation remotely.

[0085] In this embodiment, as shown in Figure 3, the controller 50 selects a candidate attachment corresponding to the image of the attachment from among a list of candidate attachments stored in the memory 50e, and outputs a guidance image to the monitor 40 based on the selected candidate attachment. This allows the controller 50 to accurately grasp the shape, dimensions, etc. of the attachment 30 from the image of the attachment 30 detected by the external sensor 20. As a result, the precise relative position between the gripping portion 8b and the first pin 31 can be displayed on the monitor 40.

[0086] Furthermore, in this embodiment, as shown in Figure 7, the guidance image includes a display indicating the target position of the arm 7 when attaching or detaching the attachment 30. Guidance is performed with the position of the arm top pin 15 as the target position. Therefore, by aligning the current position of the arm 7 with the target position of the arm 7 in the guidance image, it becomes possible to align the position of the arm top pin 15 with the target position.

[0087] In this embodiment, as shown in Figure 7, the guidance image includes a display (solid arrow in the figure) indicating a rotation instruction for the coupler 8. The rotation instruction for the coupler 8 is an instruction to rotate the coupler 8 around the arm top pin 15. As described above, after aligning the position of the arm top pin 15 to the target position, the first pin 31 can be inserted into the gripping portion 8b by rotating the coupler 8 according to the rotation instruction for the coupler 8. By operating based on the target position of the arm 7 and the rotation instruction for the coupler 8 displayed in the guidance image, it becomes possible to further improve the efficiency of attaching and detaching the attachment 30.

[0088] Furthermore, in this embodiment, as shown in Figure 9, the controller 50 obtains the relative angle between the hydraulic excavator 100 and the attachment 30 from an image of the attachment 30, and outputs a guidance image to the monitor 40 that includes instructions to operate the slewing body 3 so that the relative angle becomes smaller. As a result, the operator can easily reduce the relative angle between the hydraulic excavator 100 and the attachment 30 by following the instructions in the guidance image, thereby enabling further efficiency in attaching and detaching the attachment 30.

[0089] Furthermore, in this embodiment, as shown in Figure 9, the guidance image includes an instruction to move the traveling body 5. This makes it possible to move the hydraulic excavator 100 to a position where the coupler 8 faces the attachment 30 directly when the rotating body 3 is rotated.

[0090] <Note> The above description includes the following features.

[0091] (Note 1) A display system comprising: a sensor that acquires sensor data related to an attachment that can be attached to and detached from a coupler; a monitor; and a controller that outputs a guidance image to the monitor based on the sensor data, wherein the guidance image includes a display that allows recognition of the relative position between the gripping portion of the coupler and the pins of the attachment that can be attached to and detached from the gripping portion.

[0092] (Note 2) The display system according to Note 1, wherein the controller selects a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, and outputs the guidance image to the monitor based on the selected candidate attachment.

[0093] (Note 3) The display system described in Note 2, wherein the controller selects the candidate attachment based on information relating to the shape of the candidate attachment.

[0094] (Note 4) The coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment, as described in any one of Notes 1 to 3.

[0095] (Note 5) The guidance image is a display system according to any one of Notes 1 to 4, which includes a display indicating a rotation instruction for the coupler.

[0096] (Note 6) The display system according to any one of Notes 1 to 5, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, and the controller obtains the relative angle from the sensor data and outputs the guidance image to the monitor which includes an instruction to operate the slewing body so that the relative angle becomes smaller.

[0097] (Note 7) The coupler is attached to the work machine of a work machine having a traveling body, and the guidance image includes instructions to move the traveling body, as described in any one of Notes 1 to 5.

[0098] (Note 8) The display system according to Note 7, wherein the work machine has a rotating body to which the work machine is attached, and the guidance image includes displays indicating an instruction to move the traveling body and an instruction to rotate the rotating body as an instruction to face the coupler directly toward the attachment.

[0099] (Note 9) The display system according to Note 7, wherein the guidance image includes a display indicating an instruction to operate the traveling body such that the extending direction of the traveling body and the axial direction of the pin are perpendicular or parallel to each other, as an instruction to face the coupler directly toward the attachment.

[0100] (Note 10) The display system according to any one of Notes 1 to 9, wherein the sensor data includes an image of the attachment.

[0101] (Note 11) A display method for displaying an image on a monitor that assists in attaching an attachment to a coupler, comprising the steps of: acquiring sensor data related to the attachment; and outputting a guidance image to the monitor based on the acquired sensor data, wherein the guidance image includes a display that allows recognition of the relative position between the gripping portion of the coupler and the pins of the attachment that are attached to and detached from the gripping portion.

[0102] (Note 12) The display method according to Note 11, further comprising the step of selecting a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, wherein the guidance image is output to the monitor based on the selected candidate attachment.

[0103] (Note 13) The coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment, as described in Note 11 or Note 12.

[0104] (Note 14) The display method according to Note 11 or Note 12, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, the method further comprises the step of obtaining the relative angle from the sensor data, and the guidance image including an instruction to operate the slewing body so that the relative angle becomes smaller is output to the monitor.

[0105] (Note 15) The coupler is attached to the work machine of a work machine having a traveling body, and the guidance image includes an instruction to move the traveling body, as described in the display method of Note 11 or Note 12.

[0106] (Note 16) The display method described in any one of Notes 11 to 15, wherein the sensor data includes an image of the attachment.

[0107] (Note 17) A non-volatile storage medium storing a program that causes the controller's processor to execute the following steps: acquiring sensor data related to an attachment, and outputting a guidance image to a monitor based on the acquired sensor data, which includes a display that allows recognition of the relative positions between the gripping portion of the coupler and the pins of the attachment that are attached to and detached from the gripping portion.

[0108] (Note 18) A non-volatile storage medium according to Note 17, which stores the program that causes the processor of the controller to perform the step of selecting a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, and the guidance image is output to the monitor based on the selected candidate attachment.

[0109] (Note 19) The coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment, as described in Note 17 or Note 18, on the non-volatile storage medium.

[0110] (Note 20) The non-volatile storage medium according to Note 17 or Note 18, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, stores the program causing the processor of the controller to perform the step of obtaining the relative angle from the sensor data, and the guidance image includes instructions to operate the slewing body so that the relative angle becomes smaller.

[0111] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope of equivalents of the claims are intended to be included.

[0112] 1 Main body, 2 Work equipment, 3 Slewing body, 4 Cab, 4S Driver's seat, 5 Running body, 5Cr Tracks, 5M Travel motor, 6 Boom, 7 Arm, 8 Coupler, 8a Coupler body, 8b Gripping part, 8c Locking pin, 8d Mounting surface, 10 Boom cylinder, 11 Arm cylinder, 12 Attachment cylinder, 13 Boom foot pin, 14 Boom top pin, 15 Arm top pin, 16 Link pin, 17 Link, 20 External sensor, 21 Attitude sensor, 22 Slewing angle sensor, 23a, 23b GNSS antenna, 24 Position and direction sensor, 30 Attachment, 40 Monitor, 50 Controller, 50a Acquisition unit, 50b Selection unit, 50c Image generation unit, 50d Output control unit, 50e Memory, 60 Target position, 100 Hydraulic excavator.

Claims

1. A display system comprising: a sensor that acquires sensor data related to an attachment that can be attached to and detached from a coupler; a monitor; and a controller that outputs a guidance image to the monitor based on the sensor data, wherein the guidance image includes a display that allows recognition of the relative position between the gripping portion of the coupler and the pins of the attachment that can be attached to and detached from the gripping portion.

2. The display system according to claim 1, wherein the controller selects a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, and outputs the guidance image to the monitor based on the selected candidate attachment.

3. The display system according to claim 2, wherein the controller selects the candidate attachment based on information relating to the shape of the candidate attachment.

4. The display system according to claim 1, wherein the coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment.

5. The display system according to claim 1, wherein the guidance image includes a display indicating a rotation instruction for the coupler.

6. The display system according to claim 1, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, and the controller obtains the relative angle from the sensor data and outputs the guidance image to the monitor which includes an instruction to operate the slewing body so that the relative angle decreases.

7. The display system according to claim 1, wherein the coupler is attached to the work machine of a work machine having a traveling body, and the guidance image includes an instruction to move the traveling body.

8. The display system according to claim 7, wherein the work machine has a rotating body to which the work machine is attached, and the guidance image includes displays indicating an instruction to move the traveling body and an instruction to rotate the rotating body as an instruction to face the coupler toward the attachment.

9. The display system according to claim 7, wherein the guidance image includes a display indicating an instruction to operate the traveling body such that the extending direction of the traveling body and the axial direction of the pin are perpendicular or parallel to each other, as an instruction to face the coupler directly toward the attachment.

10. The display system according to claim 1, wherein the sensor data includes an image of the attachment.

11. A display method for displaying an image on a monitor that assists in attaching an attachment to a coupler, comprising the steps of: acquiring sensor data related to the attachment; and outputting a guidance image to the monitor based on the acquired sensor data, wherein the guidance image includes a display that allows recognition of the relative position between the gripping portion of the coupler and the pins of the attachment that are attached to and detached from the gripping portion.

12. The display method according to claim 11, further comprising the step of selecting a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, wherein the guidance image is output to the monitor based on the selected candidate attachment.

13. The display method according to claim 11, wherein the coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment.

14. The display method according to claim 11, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, the method further comprises the step of obtaining the relative angle from the sensor data, and the display method according to claim 11, wherein a guidance image including an instruction to operate the slewing body to reduce the relative angle is output to the monitor.

15. The display method according to claim 11, wherein the coupler is attached to the work machine of a work machine having a traveling body, and the guidance image includes an instruction to move the traveling body.

16. The display method according to claim 11, wherein the sensor data includes an image of the attachment.

17. A non-volatile storage medium storing a program that causes the controller's processor to execute the following steps: acquiring sensor data related to an attachment, and outputting a guidance image to a monitor based on the acquired sensor data, which includes a display that allows recognition of the relative positions between the gripping portion of the coupler and the pins of the attachment that are attached to and detached from the gripping portion.

18. The non-volatile storage medium according to claim 17, which stores a program that causes the processor of the controller to perform the step of selecting a candidate attachment corresponding to the sensor data from among a plurality of candidate attachments included in a pre-stored list, and the guidance image is output to the monitor based on the selected candidate attachment.

19. The non-volatile storage medium according to claim 17, wherein the coupler is attached to a work machine including an arm, and the guidance image includes a display indicating the target position of the arm when attaching or detaching the attachment.

20. The non-volatile storage medium according to claim 17, wherein the coupler is attached to the work machine of a work machine having a slewing body, the relative position includes the relative angle between the work machine and the attachment, stores the program causing the processor of the controller to perform the step of obtaining the relative angle from the sensor data, and the guidance image includes instructions to move the slewing body so that the relative angle decreases.

Images