Image generation system

Abstract

This image generation system (1) is used in a work machine (10) capable of automatically performing work on a work object (O) accommodated in a container (27). The image generation system (1) has a detection unit (31) and a control unit (43). The detection unit (31) detects position information of each of the container (27) and the work object (O) accommodated in the container (27). The control unit (43) determines a target path (A) for work in the container (27), and generates a confirmation image (G), which is an image for simultaneously displaying a plurality of display objects including the container (27) and / or the work object (O) and the target path (A), on the basis of the determined target path (A) and the detected position information.

Description

Image generation system

[0001] The present invention relates to an image generation system used in a work machine capable of autonomous driving.

[0002] For example, Patent Document 1 describes a work machine that performs autonomous driving. Patent Document 1 discloses displaying an image related to a generated target trajectory of autonomous driving (for example, the target trajectory of a dumping operation onto a dump truck) (for example, paragraphs 0234-0235).

[0003] As a result of the autonomous driving of the work machine based on the target trajectory, a part of the work machine (such as a front attachment) may interfere with a container such as the loading platform of a dump truck. In addition, as a result of a part of the work machine interfering with a work object such as earth and sand in the container, unnecessary pressure may be generated on the container. Therefore, it is desired to generate an image for preliminarily checking the influence of the autonomous driving of the work machine based on the target trajectory on the container.

[0004] Japanese Patent Publication No. 2021-054436

[0005] Therefore, an object of the present invention is to provide an image generation system capable of generating an image for checking the influence of the autonomous driving of a work machine on a container.

[0006] The image generation system is used in a work machine capable of performing work on a work object accommodated in a container by autonomous driving. The image generation system includes a detection unit that detects the position information of each of the container and the work object accommodated in the container, and determines a target path of the work in the container. Based on the determined target path and the position information detected by the detection unit, a control unit generates a confirmation image, which is an image that simultaneously displays a plurality of display targets including the container and / or the work object and the target path.

[0007] With the above image generation system, an image for checking the influence of the work of a work machine performing autonomous driving on a container can be generated.

[0008] This is a diagram showing the work machine 10 and the image generation system 1. This is a block diagram of the image generation system 1 shown in Figure 1. This is a diagram showing an example of a confirmation image G generated by the image generation system 1 shown in Figure 1. This is a diagram showing an example of a confirmation image G generated by the image generation system 1 shown in Figure 1. This is a diagram showing an example of a confirmation image G generated by the image generation system 1 shown in Figure 1. This is a flowchart of the image generation process.

[0009] The image generation system 1 will be described with reference to the drawings.

[0010] The image generation system 1 shown in Figure 1 is a system that generates a confirmation image G (see Figure 3) which is an image that simultaneously displays multiple display objects, including at least one of the work object O that is the target of the work machine 10, the container 27 that contains the work object O, and the target path A of the work machine 10. The image generation system 1 comprises a detection unit 31 (see Figure 2) and an information processing device 40.

[0011] The work machine 10 is a machine that performs work. The work machine 10 may also be a construction machine that performs construction work. The work machine 10 may be, for example, an excavator or a crane. The work machine 10 is configured to be operable by automatic control. The automatic control may be fully automatic operation or semi-automatic operation (machine control). The work machine 10 is configured to perform work on the work object O contained in the container 27 by automatic operation. The work machine 10 may also operate in response to the operation of a worker (operator) without the use of automatic control. For example, the work machine 10 may be operated (onboard operation) by a worker in the driver's cab 13a (described later), or it may be remotely operated from outside the work machine 10. The following mainly describes the case where the work machine 10 is an excavator.

[0012] The work machine 10 comprises a machine body 10a, an attachment 15, a drive control unit 17 (see Figure 2), and an actuator 21 (see Figure 2).

[0013] The machine body 10a is the main body of the work machine 10. The machine body 10a comprises a lower body 11 and an upper slewing body 13. The lower body 11 supports the upper slewing body 13 so that it can rotate. As shown in the example in Figure 1, the lower body 11 may be a lower traveling body that can travel on a traveling surface (such as the ground). If the lower body 11 is traveling, it may be equipped with crawlers or wheels. The upper slewing body 13 is supported by the lower body 11 so that it can rotate. An attachment 15 is attached to the upper slewing body 13. The upper slewing body 13 includes a driver's cab 13a. The driver's cab 13a is a room in which an operator who operates the work machine 10 sits. When the work machine 10 operates in response to operator operation, the work machine 10 may be operated (operated by the operator) inside the driver's cab 13a, or it may be remotely operated from outside the work machine 10.

[0014] The direction in which the axis of rotation (center of rotation) of the upper slewing body 13 extends relative to the lower body 11 is defined as the vertical direction Z. In the vertical direction Z, the side extending from the lower body 11 toward the upper slewing body 13 is defined as the upper side Z1, and the side opposite to the upper side Z1 is defined as the lower side Z2. The direction in which the axis of rotation of the attachment 15 (boom 15a, described later) extends relative to the upper slewing body 13 is defined as the horizontal direction Y. The direction perpendicular to both the vertical direction Z and the horizontal direction Y is defined as the front-rear direction X. In the front-rear direction X, the side on which the attachment 15 protrudes relative to the upper slewing body 13 is defined as the rear side X1, and the side opposite to the rear side X1 is defined as the front side X2.

[0015] Attachment 15 is the part that performs the work. Attachment 15 performs work on the object to be worked on O. Attachment 15 is attached to the machine body 10a (more specifically, the upper slewing body 13). For example, attachment 15 comprises a boom 15a, an arm 15b, and a tip attachment 15c. The boom 15a is rotatably (up and down) attached to the upper slewing body 13 around an axis extending in the lateral direction Y. The arm 15b is rotatably attached to the boom 15a around an axis extending in the lateral direction Y.

[0016] The tip attachment 15c is provided at the tip of the attachment 15. The tip attachment 15c is rotatably mounted on the arm 15b around an axis extending in the lateral direction Y. The tip attachment 15c may be a bucket capable of scooping up and excavating the workpiece O. The tip attachment 15c may be equipped with a device for gripping the workpiece O (grapple, nibbler, rotating fork, etc.), a device for crushing the workpiece O (breaker, etc.), or a magnet for attracting the workpiece O made of a magnetic material (metal, etc.). The following description will mainly focus on the case where the work machine 10 is equipped with a bucket as the tip attachment 15c.

[0017] The work machine 10 may also have a dozer as a work device for performing work. The dozer is a work device attached to the lower body 11. The dozer includes, for example, a plate-shaped member (e.g., a dozer blade) extending in the lateral direction Y (width direction) and vertical direction Z of the lower body 11. The dozer may also be movable in the vertical direction Z relative to the lower body 11.

[0018] The work object O is the object that the work machine 10 will operate on. The work object O may be soil, rock, magnetic material (such as metal), resin, waste, wood (such as logs), or structure (such as blocks). If the work object O is soil, it may be in the form of soil, granules, chips, powder, etc.

[0019] The actuator 21 (see Figure 2) is a device that moves the work machine 10. The actuator 21 may be a hydraulic actuator that operates using hydraulic pressure, or an electric actuator that operates using electricity. The actuator 21 may be a motor that generates rotational motion, or a linear actuator (for example, an extendable cylinder) that generates linear motion. The actuator 21 may be a travel motor that moves the work machine 10. The actuator 21 may be a slewing motor that rotates the upper slewing body 13 relative to the lower body 11. The actuator 21 may be a cylinder that moves the attachment 15.

[0020] The drive control unit 17 (see Figure 2) controls the actuator 21 that moves the work machine 10. The drive control unit 17 may also include a hydraulic circuit that controls a hydraulic actuator that operates using hydraulic pressure. The drive control unit 17 may also include an electrical circuit that controls an electric actuator that operates using electric power.

[0021] The container 27 contains the object to be worked on O. The container 27 is, for example, a box shape without a lid. The container 27 only needs to be capable of containing the object to be worked on O, and its form is not particularly limited. For example, the container 27 may be the cargo bed of a vehicle 25 (see Figure 1), or it may be something placed directly on the ground (for example, a soil pit). At least a part of the container 27 may be made up of the ground. The following description will mainly focus on the case where the container 27 is the cargo bed of a vehicle 25.

[0022] The vehicle 25 comprises a vehicle body 26 and the container 27 described above. The vehicle body 26 supports the container 27 (cargo bed). The vehicle body 26 is drivable and may be driven by wheels or by crawlers. The vehicle body 26 includes a vehicle driver's cab 26a. The container 27 comprises a container floor 27a, a container rear 27b, a container side 27c, and a container front 27d. The container 27 is positioned behind the vehicle driver's cab 26a. The container 27 may be movable relative to the vehicle body 26 or fixed to the vehicle body 26. The following describes the case where the container floor 27a, which is the bottom surface of the container 27, is positioned horizontally or substantially horizontally.

[0023] The directions of the container 27 will now be explained. As shown in Figure 1, the longitudinal direction of the container 27 when it is placed on a horizontal plane is defined as the container front-rear direction U. One side in the container front-rear direction U is defined as the container front side U1, and the side opposite to the container front side U1 is defined as the container rear side U2. For example, the container front side U1 is the side facing the vehicle driver's cab 26a from the container 27, and the container rear side U2 is the side facing the container 27 from the vehicle driver's cab 26a. The direction perpendicular to the container front-rear direction U and along the container floor 27a is defined as the container lateral direction V. When the container 27 is placed on a horizontal plane, the vertical direction Z is perpendicular or approximately perpendicular to the container front-rear direction U and the container lateral direction V, respectively.

[0024] The container base portion 27a is the part (bottom) that constitutes the lower wall Z2 of the container 27. The container rear portion 27b is the part that constitutes the rear wall U2 of the container 27. The container rear portion 27b protrudes upward Z1 from the end of the rear wall U2 of the container base portion 27a and has a plate-like shape, for example. The container rear portion 27b is provided along a plane that extends perpendicular or approximately perpendicular to the front-rear direction U of the container (the same applies to the container front portion 27d). The container side portion 27c is the part that constitutes the outer wall of the container 27 in the lateral direction V. The container side portion 27c protrudes upward Z1 from the outer ends (left and right ends) of the container base portion 27a in the lateral direction V of the container and has a plate-like shape, for example. The container side portion 27c is provided along a plane that extends perpendicular or approximately perpendicular to the lateral direction V of the container. The container front portion 27d is the part that constitutes the front wall U1 of the container 27. The front portion 27d of the container protrudes upward Z1 from the end of the front side U1 of the container floor portion 27a, and for example, it has a plate-like shape. If the container 27 is a loading platform, the front portion 27d of the container protrudes upward Z1 above the side portion 27c and the rear portion 27b of the container.

[0025] The detection unit 31 (see Figure 2) detects various states. Part or all of the detection unit 31 may be mounted on the work machine 10 or located outside the work machine 10. The same applies to the information processing device 40, which will be described later, that it may be mounted on the work machine 10 or located outside the work machine 10 (for example, at the work site). The detection unit 31 detects the position information of the container 27. The detection unit 31 detects the position information of the work object O contained in the container 27.

[0026] As shown in Figure 2, the detection unit 31 includes a position detection unit 311, a direction detection unit 312, an imaging device 313, and an attitude detection unit 315. The position detection unit 311 detects the position of the object to be measured. The position detection unit 311 detects the position of a specific part of the work machine 10 (see Figure 1). For example, the position detection unit 311 may detect the position of a specific part of the upper rotating body 13, or the position of a specific part of the attachment 15. The position detection unit 311 may include a device that detects position using electromagnetic waves (light, radio waves, etc.). The position detection unit 311 may include a device that uses a satellite positioning system, for example, a device that uses GNSS (Global Navigation Satellite System). The position detection unit 311 may include a device that detects position without using satellites, a device that detects position using a ground transmitter and receiver, or a device that detects position using the reflection of light (e.g., laser light) (e.g., a total station). The position detection unit 311 may calculate the position of the object to be measured based on position information detected by multiple types of devices.

[0027] The direction detection unit 312 detects the direction (orientation, posture) of the object to be measured. The direction detection unit 312 detects the direction of a specific part of the work machine 10. For example, the direction detection unit 312 may detect the direction of a specific part of the upper rotating body 13, or it may detect the direction of a specific part of the attachment 15. The direction detection unit 312 may be equipped with a device that uses the Earth's magnetic field to detect the orientation of the object to be measured. The direction detection unit 312 may also detect the direction of the object to be measured based on the positions of multiple parts of the object to be measured relative to the work site (for example, positions detected by the position detection unit 311).

[0028] The imaging device 313 images the object to be imaged. The imaging device 313 may image part or all of the work machine 10, or it may image objects around the work machine 10. The imaging device 313 detects three-dimensional information of the object to be imaged, for example, by acquiring a three-dimensional depth image of the object to be imaged. The three-dimensional information is a type of positional information as described above, and includes information that identifies the position in three-dimensional space, such as the distance (depth), angle (orientation), and height of the object to be imaged. The imaging device 313 may detect not only three-dimensional information but also two-dimensional information (two-dimensional images).

[0029] The imaging device 313 may be passive or active. Specifically, the imaging device 313 may be equipped with a camera (monocular camera) that detects two-dimensional information. The imaging device 313 may be equipped with a stereo camera that detects three-dimensional information. The imaging device 313 may detect three-dimensional information of an object by irradiating the object with waves such as electromagnetic waves and detecting the reflected waves. The imaging device 313 may be equipped with a Time of Flight (TOF) sensor that detects distance based on the time from wave irradiation to the return of the reflected wave, or it may be equipped with a sensor that detects distance based on the frequency of the reflected wave. The imaging device 313 may be equipped with a device that detects three-dimensional information using light (e.g., laser light), for example, LiDAR (Light Detection and Ranging). The imaging device 313 may be equipped with a device that detects three-dimensional information using radio waves (e.g., millimeter-wave radar).

[0030] The imaging device 313 may be provided as a single unit or as a group. If multiple imaging devices 313 are provided, the types of imaging devices 313 (type of imaging method, whether two-dimensional or three-dimensional, etc.) may be the same or different. The imaging device 313 may detect three-dimensional information of the object to be imaged by combining multiple types of information (for example, two-dimensional and three-dimensional information).

[0031] The posture detection unit 315 detects the posture of the work machine 10. The posture detection unit 315 may also detect the position and orientation of the work machine 10 relative to the work site. The posture detection unit 315 may also detect the position and orientation of a reference part of the work machine 10 relative to the work site. The reference part of the work machine 10 is, for example, a specific part of the upper slewing body 13 or the lower body 11. The reference part of the work machine 10 may be the attachment point (boom foot) of the boom 15a to the upper slewing body 13, or a specific part on the pivot axis of the upper slewing body 13 relative to the lower body 11. The posture detection unit 315 may detect information (angle, angular velocity, angular acceleration, etc.) about the rotation of the upper slewing body 13 relative to the lower body 11. The posture detection unit 315 may also detect information (angle, angular velocity, angular acceleration, etc.) about the rotation of the boom 15a relative to the upper slewing body 13. The posture detection unit 315 may also detect information about the rotation of the arm 15b relative to the boom 15a. The attitude detection unit 315 may also detect information about the rotation of the tip attachment 15c relative to the arm 15b.

[0032] The information processing device 40 is a device that processes information related to the work machine 10. The information processing device 40 is a device that processes information related to the automatic operation of the work machine 10. The information processing device 40 has a function to remotely control the work machine 10 that is capable of automatic operation. The information processing device 40 may also have a function to support the management of the work machine 10 that is capable of automatic operation. The information processing device 40 is connected to the work machine 10 in a communicative manner. Hereinafter, a communicative connection will also be simply referred to as "connection". The information processing device 40 may be equipped with various information processing devices. The functions of the information processing device 40 may be distributed and arranged across multiple information processing devices. In other words, the information processing device 40 may be a so-called distributed system.

[0033] The information processing device 40 includes, for example, a server 40a (computer) and an information terminal 40b. The server 40a and the information terminal 40b may be connected by wireless communication or by wired communication. For example, communication may be performed by means of a mobile phone line, optical fiber line, wireless LAN (Local Area Network), or wired LAN.

[0034] For example, server 40a is a computer (controller) that performs information input / output, calculations (processing), and information storage. For example, the functions of server 40a are realized by the execution of a program stored in server 40a by the calculation unit. Server 40a is located outside the work machine 10 and the information terminal 40b. Server 40a is connected to the work machine 10. Information terminal 40b is connected to the work machine 10. Information terminal 40b may be connected to the work machine 10 via server 40a, or it may be connected to the work machine 10 without going through server 40a.

[0035] The information terminal 40b is a computer capable of inputting and outputting information. The information terminal 40b may or may not be portable. As shown in Figure 1, in this embodiment, the information terminal 40b is a tablet, but it may also be a smartphone or a personal computer. The information terminal 40b may also be used in conjunction with the server 40a. The information terminal 40b may be located outside the work machine 10, or it may be located inside the work machine 10 (for example, it may be brought in).

[0036] As shown in Figure 2, the information processing device 40 includes an input unit 41, a display unit 42, a control unit 43, and a storage unit 44. The input unit 41 is a device for inputting information (input device). The input unit 41 accepts operations from the operator. For example, the input unit 41 is operated by the operator and outputs a signal corresponding to the operation. The input unit 41 outputs information to the control unit 43. In this embodiment, the input unit 41 is a touch panel provided on the information terminal 40b (see Figure 1). The input unit 41 may be, for example, a mouse, a keyboard, a device that inputs information based on the operator's line of sight, or a device that inputs information based on the position information of the operator's fingers in space. The input unit 41 may also be equipped with a device for voice input (specifically, a microphone). The input unit 41 may be provided on a smartphone or on a personal computer. The operator can input information related to automatic driving through operations on the input unit 41. The operator can input information for remotely controlling the work machine 10 through operations on the input unit 41.

[0037] The display unit 42 is a device that outputs information. The display unit 42 displays information related to the work machine 10. The display unit 42 displays information related to automatic operation. The display unit 42 outputs information based on signals output from the control unit 43. As shown in Figure 1, in this embodiment, the display unit 42 is a monitor (display) provided on the information terminal 40b, but it may be provided on a smartphone or a personal computer. The display unit 42 may be equipped with a projection device that projects onto an object such as the ground. The display unit 42 may be equipped with a light-emitting device (light). The display unit 42 may display VR images using VR (Virtual Reality) technology, or display AR images using AR (Augmented Reality) technology. The display unit 42 may change at least one of the hue, density (transparency), brightness, and saturation of the light it outputs. The image displayed by the display unit 42 (the image that the control unit 43 causes the display unit 42 to display) is called the display image. The displayed image includes shapes, patterns, colors, characters, symbols, etc. The displayed image also includes a GUI (Graphical User Interface). The information terminal 40b may also be equipped with functions for outputting information other than display, such as an audio output unit and a vibration output unit.

[0038] The control unit 43 is a computer (controller) that performs signal input / output and calculations (processing). For example, the functions of the control unit 43 are realized by the control unit 43 executing a program stored in the storage unit 44. The control unit 43 processes information related to the display unit 42, for example, causing the display unit 42 to display information. The control unit 43 processes information input to the input unit 41. The control unit 43 may also perform other controls, such as information communication control. The control unit 43 and the storage unit 44 may be provided on the server 40a or on the information terminal 40b, respectively. The functions of the control unit 43 and the storage unit 44 may be distributed and provided on the server 40a and the information terminal 40b.

[0039] The control unit 43 includes a container position recognition unit 431, a storage position recognition unit 432, a target path generation unit 433, a confirmation image generation unit 434, and an automatic driving control unit 435.

[0040] The container position recognition unit 431 acquires the position information of the container 27 (see Figure 1) based on the information detected by the detection unit 31. The container position recognition unit 431 transmits the acquired position information, such as the coordinates of the container 27, to the confirmation image generation unit 434.

[0041] The storage location recognition unit 432 acquires location information of the work object O stored in the container 27 based on the information detected by the detection unit 31. For example, the acquired location information may be coordinate information indicating the position of the work object O inside the container 27, or it may be location information identified from an image (e.g., a three-dimensional image) of the work object O inside the container 27. The container location recognition unit 431 transmits the location information of the work object O stored in the container 27 to the confirmation image generation unit 434.

[0042] The target path generation unit 433 determines the target path A (see Figure 1) for the work to be performed on the work object O contained in the container 27. For example, based on the position (coordinates) of the container 27 and the position (coordinates) of the work object O, the target path generation unit 433 determines a target path A that does not interfere with the container 27 but does not interfere with the work object O. When setting the target path A, if the coordinate systems of the objects are different, the coordinate systems are unified.

[0043] The confirmation image generation unit 434 generates a confirmation image G (see Figures 3-5) which is an image that simultaneously displays the container 27 and / or the work object O and the target path A, based on at least one of the location information of the container 27 acquired by the container location recognition unit 431 and the location information of the work object O acquired by the storage location recognition unit 432, and the target path A determined by the target path generation unit 433.

[0044] The automatic operation control unit 435 automatically controls the work machine 10 so that the work machine 10 moves according to the work plan. The automatic operation control unit 436 outputs a command to the drive control unit 17 so that the work machine 10 moves according to the work plan. The automatic operation control unit 436 controls the movement of the work machine 10 based on the posture detected by the posture detection unit 315. For example, the automatic operation control unit 436 starts the automatic operation of the work machine 10 based on the operation of the operator received by the input unit 41. Details of the automatic operation will be described later.

[0045] The storage unit 44 stores information. For example, the storage unit 44 stores programs and data. For example, the storage unit 44 stores the work plan. Also, for example, the storage unit 44 stores the dimensions of the attachment 15 including the tip attachment 15c (such as a bucket).

[0046] (Automatic operation of the work machine 10) The work machine 10 shown in FIG. 1 is configured to be able to perform automatic operation based on a work plan. The work plan is information regarding the plan of the work of the work machine 10 and is set in advance (before automatic operation) in the image generation system 1. The work plan may be set by the target path generation unit 433 (see FIG. 2). The work plan may include information on the target route of the travel of the work machine 10. The work plan may include information on the target range where the tip attachment 15c performs work.

[0047] (Target Path A) The work plan includes information on the above-described target path A (target locus) that defines the path of the specific part 15c1 of the attachment 15. The specific part 15c1 of the attachment 15 may be, for example, the tip of the tip attachment 15c or the base end part of the tip attachment 15c (the tip part of the arm 15b). In the example of FIG. 1, the specific part 15c1 is the tip of the tip attachment 15c. The specific part 15c1 of the attachment 15 may be at one location or at a plurality of locations. The above "target path A" is information including, for example, information (coordinates) on the positions of a plurality of target points and information on the order of each target point. The target path A is set between the start position A1 and the end position A2. The start position A1 is the starting point of the target path A, and the end position A2 is the ending point of the target path A. The start position A1 and the end position A2 are set within the reachable range of the tip attachment 15c.

[0048] The target path A may be a target path for work limited to within the container 27 or a target path for work passing through the inside of the container 27. For example, the target path A may be a target path for an operation (e.g., excavation work) to capture the work object O within the container 27 or a target path for an operation (e.g., soil discharge work) to release the work object O into the container 27. Also, for example, the target path A may be a target path for a leveling operation to level the work object O loaded in the container 27 with the tip attachment 15c (e.g., a bucket). Hereinafter, mainly, the case where the work in the container 27 is a leveling operation will be described. The target path A may have time information.

[0049] (Regarding leveling work) The work machine 10 leveles the work object O, that is, makes the top of the work object O flat, thereby preventing the work object O from collapsing. For example, when the work object O is transported in a container 27, the leveling of the work object O prevents it from collapsing and prevents it from spilling out of the container 27. For example, when the work object O is placed on the ground, the leveling of the work object O prevents it from collapsing due to its own weight or wind. For example, compared to soil removal work, leveling work is more likely to bring the tip attachment 15c closer to the container 27. Therefore, in leveling work, the tip attachment 15c is more likely to interfere with the container 27, and the worker needs to monitor it more carefully.

[0050] The control unit 43 (see Figure 2) sets a target path A for the leveling work based on, for example, the three-dimensional information of the object to be imaged detected by the imaging device 313 (see Figure 2). The target path A for the leveling work is the path that a specific part 15c1 of the tip attachment 15c passes through when the leveling work is performed. The target path A is the path from the starting position A1 where the leveling work begins to the ending position A2 where the leveling work ends. The control unit 43 causes the work machine 10 to perform the leveling work by controlling the work machine 10 so that the specific part 15c1 of the tip attachment 15c moves along the target path A. The control unit 43 may also control the work machine 10 so that the specific part 15c1 of the tip attachment 15c moves linearly from the starting position A1 to the ending position A2. The target path A does not have to be a straight line. For example, the target path A may be staircase-shaped or mountain-shaped in a side view. If the target path A is not a straight line, the control unit 43 may set one or more intermediate positions between the start position A1 and the end position A2 when setting the target path A.

[0051] The control unit 43 controls the drive control unit 17 (see Figure 2) to cause the work machine 10 to perform leveling work. The control unit 43 causes the work machine 10 to perform leveling work by controlling at least one of the operation of the attachment 15 and the rotation of the upper rotating body 13 relative to the lower body 11. Alternatively, the control unit 43 may cause the work machine 10 to perform leveling work by controlling the movement of the lower body 11. In the leveling work, the direction in which the tip attachment 15c moves, that is, the leveling direction of the tip attachment 15c, may be the rear side X1 or the front side X2. In other words, the leveling work may be pushing leveling or pulling leveling. The leveling direction of the tip attachment 15c may be the same as the rotation direction of the upper rotating body 13 relative to the lower body 11. The leveling direction of the tip attachment 15c may be the container front-rear direction U, a direction inclined with respect to the container front-rear direction U, or the container side direction V.

[0052] The leveling operation may be performed only once. After the first leveling operation is completed, subsequent leveling operations may be performed. For example, if there is a difference of a predetermined amount or more between the height of the actual work object O and the height of the target leveling surface, the control unit 43 may control the automatic operation to perform the leveling operation multiple times at positions with different heights. For example, if the control unit 43 performs the leveling operation twice, the control unit 43 may perform the second leveling operation at a lower position than the first leveling operation.

[0053] The control unit 43 may change the height of the tip attachment 15c while moving it from the starting position A1 to the ending position A2 during the leveling operation. For example, if the pressure on the tip attachment 15c increases while it is moving, the control unit 43 may raise the position of the tip attachment 15c during the leveling operation to avoid the pressure. For example, the control unit 43 may monitor the pressure of the actuator 21 (e.g., a cylinder) during the leveling operation and raise the position of the tip attachment 15c if the pressure increases.

[0054] (Regarding the confirmation image G) The confirmation image G (see Figures 3-5) is generated by the confirmation image generation unit 434 (see Figure 2) of the control unit 43. As described above, the confirmation image G is an image that simultaneously displays multiple display objects, including the container 27 and / or the work object O and the target path A. The confirmation image G includes at least one view image GE of the display objects, i.e., the container 27 and / or the work object O and the target path A, viewed from a specific direction (viewpoint). That is, the confirmation image G may include only one view image GE of the display objects viewed from one direction (viewpoint), or it may include multiple view images GE of the display objects viewed from multiple different directions (viewpoints). For example, the confirmation image G may include a first view image GEa (see Figures 3-5), which is a top view of the display objects, or a second view image GEb (see Figures 3-5), which is a side view of the display objects. Furthermore, the view image GE may be an image viewed from a specific direction other than a top view or side view (for example, an image viewed from the back).

[0055] In the confirmation image G, the images of the container 27 and the workpiece O may be point cloud images obtained by imaging point cloud data based on three-dimensional information, or modeled images obtained by imaging a model generated from the point cloud data. A point cloud image is an image that represents the shape using a collection of many points corresponding to the point cloud data, while a modeled image is an image that represents the shape using the contour lines of a model generated from the point cloud data. The model may be automatically generated from the point cloud data according to a specific modeling procedure, or it may be generated by manually selecting one modeling procedure from a plurality of pre-prepared modeling procedures.

[0056] The confirmation image G may be displayed on the display unit 42 (see Figure 2). This allows the operator to visually confirm the target path A for the work on the work object O contained in the container 27 through the display unit 42. In addition, the confirmation image G may be subjected to image recognition processing by an image recognition function unit (not shown). For example, the image recognition function unit may determine the impact on the container 27 due to the automatic operation of the work machine 10 by performing image recognition processing on the confirmation image G. Such an image recognition function unit may be provided inside or outside the image generation system 1.

[0057] The generation of the confirmation image G described above makes it easier to determine whether the target path A in autonomous driving is set correctly. For example, it becomes easier to determine whether the target path A is such that the tip attachment 15c presses too hard against the container 27 via the work object O.

[0058] (Regarding the view image GE) As described above, the view image GE is an image in which the container 27 and / or the workpiece O and the target path A are displayed simultaneously. In the view image GE, the target path A may be displayed overlapping with the container 27 and / or the workpiece O. The view image GE may further include the trajectory range of the tip attachment 15c (e.g., bucket). The trajectory range is the trajectory of the tip attachment 15c when it moves. The trajectory range is determined based on the dimensions of the tip attachment 15c and the target path A. That is, the control unit 43 may determine the trajectory range of the bucket based on the dimensions of the tip attachment 15c (e.g., bucket) and the target path A. This trajectory range may be displayed as the target path A, or the trajectory range may be displayed in addition to the target path A. The control unit 43 may also further display the range of the tip attachment 15c at the starting position A1 of the target path A (see Figure 1). Furthermore, the control unit 43 may display the range of the tip attachment 15c at the end position A2 of the target path A (see Figure 1). Also, the control unit 43 may display an image showing the direction from the start position A1 to the end position A2 of the target path A.

[0059] (Specific example of confirmation image G) [Display example 1] Figure 3 shows display example 1 of confirmation image G. As shown in this figure, confirmation image G in display example 1 includes a first view image GEa, which is an image of the display target viewed from above, and a second view image GEb, which is an image of the display target viewed from the side.

[0060] The first view image GEa is an image that simultaneously displays the container 27 and the target path A in a top view; in other words, it is an image that shows the positional relationship between the container 27 and the target path A in a top view. The first view image GEa includes a container top view image Ga27, a target path top view image GaA, a start position top view image GaA1, and an end position top view image GaA2.

[0061] The container top image Ga27 is an image showing the container 27 from a top view. In display example 1, the container top image Ga27 is a modeled image of the container 27 generated based on its three-dimensional information, viewed from above. The container top image Ga27 comprises a floor top image Ga27a, a rear top image Ga27b, two side top images Ga27c, and a front top image Ga27d. The floor top image Ga27a is an image of the model of the container floor 27a viewed from above. The rear top image Ga27b is an image of the model of the container rear 27b viewed from above. The side top image Ga27c is an image of the model of the container side 27c viewed from above. The front top image Ga27d is an image of the model of the container front 27d viewed from above. In the container top view image Ga27, the rear top view image Ga27b, two side top view images Ga27c, and the front top view image Ga27d are arranged to surround the rectangular floor top view image Ga27a.

[0062] The target path top view image GaA is an image that displays the trajectory range of the tip attachment 15c as it moves along the target path A, viewed from above. The starting position top view image GaA1 is an image that displays the range of the tip attachment 15c at the starting position A1, viewed from above. The ending position top view image GaA2 is an image that displays the range of the tip attachment 15c at the ending position A2, viewed from above. The target path top view image GaA, the starting position top view image GaA1, and the ending position top view image GaA2 are displayed superimposed on the container top view image Ga27 (floor top view image Ga27a).

[0063] The second view image GEb is an image that simultaneously displays the container 27, the workpiece O contained in the container 27, and the target path A in a side view. The second view image GEb includes a container side view image Gb27, a target path side view image GbA, a start position side view image GbA1, an end position side view image GbA2, and a workpiece side view image GbO.

[0064] The container side view image Gb27 is an image that displays the container 27 in a side view. In the case of display example 1, the container side view image Gb27 is a modeled image that shows a side view of a model generated based on the three-dimensional information of the container 27. The container side view image Gb27 comprises a floor side view image Gb27a, a rear side view image Gb27b, and a front side view image Gb27d. The floor side view image Gb27a is an image that shows a side view of the model of the container floor 27a. The rear side view image Gb27b is an image that shows a side view of the model of the container rear 27b. The front side view image Gb27d is an image that shows a side view of the model of the container front 27d. In the container side view image Gb27, the rear side view image Gb27b, the floor side view image Gb27a, and the front side view image Gb27d are arranged to form a generally shallow U-shape as a whole.

[0065] The target path side view image GbA is an image that displays the target path A of the tip attachment 15c in a side view. In display example 1, the target path side view image GbA is an image of an arrow. The direction of the arrow in the target path side view image GbA is the direction from the starting position A1 to the ending position A2. Thus, the target path side view image GbA is an image that shows the direction from the starting position A1 to the ending position A2. The starting position side view image GbA1 is an image that displays the range of the tip attachment 15c at the starting position A1 in a side view. The ending position side view image GbA2 is an image that displays the range of the tip attachment 15c at the ending position A2 in a side view.

[0066] The work object side view image GbO is an image that displays the work object O in a side view. In display example 1, the work object side view image GbO is a modeled image that displays a side view of a model generated based on the three-dimensional information of the work object O. Specifically, the work object side view image GbO includes a height image GbOa and a contour image GbOb. The height image GbOa is an image of multiple plots indicating the height positions of each part of the work object O detected by the detection unit 31 at intermittent positions. The contour image GbOb is an image of lines that smoothly connect the multiple plots of the height image GbOa.

[0067] [Display Example 2] Figure 4 shows Display Example 2 of the confirmation image G. As shown in this figure, in Display Example 2, similar to Display Example 1 (see Figure 3) above, the confirmation image G includes a first view image GEa, which is a top view of the display object, and a second view image GEb, which is a side view of the display object. Each view image GEa and GEb displays images showing the container 27, the work object O, and the target path A. The images of the work object O and the target path A displayed in Display Example 2 are the same as those in Display Example 1. On the other hand, in Display Example 2, unlike Display Example 1, a combination of a modeled image and a point cloud image is used as the image of the container 27.

[0068] In other words, in Display Example 2, the container top image Ga27 is an image displayed by superimposing a modeled image of a model generated based on the three-dimensional information of the container 27, viewed from above, and a point cloud image of a point cloud (point cloud data) that represents the three-dimensional information as a collection of many points, viewed from above. Similarly, the container side image Gb27 is an image displayed by superimposing a modeled image of the above model viewed from the side, and a point cloud image of the above point cloud (point cloud data) viewed from the side. Thus, in Display Example 2, the confirmation image G includes a point cloud image that represents the three-dimensional information of the container 27 as a point cloud. Therefore, even if the model of the container 27 generated based on the three-dimensional information deviates from the actual shape of the container 27, it is possible to generate a confirmation image G that allows confirmation of the impact of the work on the container 27.

[0069] In Display Example 2, as in Display Example 1, the modeled image is displayed as a side view image GbO of the work object O. However, similar to the image of the container 27 described above, the side view image GbO of the work object may be displayed by overlaying the point cloud image of the work object O onto the modeled image, or only the point cloud image may be displayed as the side view image GbO of the work object.

[0070] [Display Example 3] Figure 5 shows Display Example 3 of the confirmation image G. As shown in this figure, in Display Example 3, similar to Display Example 1 (see Figure 3) above, the confirmation image G includes a first view image GEa, which is a top view of the display object, and a second view image GEb, which is a side view of the display object. Images showing the container 27, the work object O, and the target path A are displayed in each view image GEa and GEb. The image of the target path A displayed in Display Example 3 is the same as that in Display Example 1. On the other hand, the images of the container 27 and the work object O displayed in Display Example 3 are different from those in Display Example 1. For example, in Display Example 3, a combination of a modeled image and a point cloud image is used as the image of the container 27 (similar to Display Example 2), and a point cloud image is used as the image of the work object O.

[0071] Specifically, in Display Example 3 shown in Figure 5, the container top image Ga27 and the container side image Gb27 are images created by overlaying a modeled image of the container 27 with a point cloud image of the container 27, respectively. The workpiece side image GbO is a point cloud image representing the three-dimensional information of the workpiece O. Thus, in Display Example 3, since the images showing the container 27 and the workpiece O include point cloud images, the loss of information due to converting three-dimensional images to modeled images can be reduced.

[0072] Furthermore, unlike Display Examples 1 and 2, Display Example 3 also displays a point cloud image of the work machine 10. Specifically, in Display Example 3, a top view image Ta of the work machine, which is a top view of the point cloud (point cloud data) representing the three-dimensional information of the work machine 10, is displayed in the first view image GEa, and a side view image Tb of the work machine, which is a side view of the point cloud, is displayed in the second view image GEb. Also, unlike Display Examples 1 and 2, in Display Example 3, parts of the vehicle 25 other than the container 27 are displayed by point cloud images. However, these parts may be displayed by modeled images, as in Display Examples 1 and 2.

[0073] (Image Generation Process) Next, the image generation process method and image generation process program of the image generation system 1 will be described. Specifically, an example of the operation of the image generation system 1 by the control unit 43 (see Figure 2) will be described with reference to Figure 6, which is a flowchart of the image generation process.

[0074] First, the control unit 43 starts automatic operation to perform leveling work (step S10). Then, the control unit 43 generates a target path A for the work inside the container 27 (step S20).

[0075] Then, the control unit 43 generates visualization data for displaying the generated target path A on the confirmation image G (step S30). For example, the control unit 43 generates data for displaying the target path side view image GbA (see Figures 3-5) on the second view image GEb.

[0076] Then, the control unit 43 generates visualization data of the trajectory range of the tip attachment 15c corresponding to the target path A (step S40). For example, the control unit 43 obtains data such as the dimensions of the tip attachment 15c from the storage unit 44, and based on the obtained data and the target path A, generates data for displaying the target path top surface image GaA (see Figures 3-5) on the first view image GEa.

[0077] Then, the control unit 43 generates visualization data for displaying the container 27 on the confirmation image G (step S50). For example, the control unit 43 acquires three-dimensional information of the container 27 from the detection unit 31 and generates visualization data of the container 27 based on the acquired three-dimensional information. The visualization data is, for example, data for displaying the container top image Ga27 and the container side image Gb27 on the confirmation image G (first and second view images GEa, GEb), and may be point cloud data of the container 27, or a model of the container 27 generated from the point cloud data.

[0078] Then, the control unit 43 generates visualization data for displaying the work object O on the confirmation image G (step S60). For example, the control unit 43 acquires three-dimensional information of the work object O from the detection unit 31 and generates visualization data of the work object O based on the acquired three-dimensional information. The visualization data is, for example, data for displaying a side view image GbO of the work object on a second view image GEb, and may be point cloud data of the work object O, or a model of the work object O generated from the point cloud data.

[0079] Then, the control unit 43 displays a confirmation image G (see Figures 3-5) on the display unit 42, which includes an image showing the container 27 and / or the work object O, and an image showing the target path A (step S70). For example, the control unit 43 generates a target path side image GbA based on the visualization data of the target path A generated in step S30, generates a target path top image GaA based on the visualization data of the trajectory range generated in step S40, generates a container top image Ga27 and a container side image Gb27 based on the visualization data of the container 27 generated in step S50, and generates a work object side image GbO based on the visualization data generated in step S60. Then, the control unit 43 displays the confirmation image G (first and second view images GEa, GEb) shown in Figures 3-5 by combining these generated images and outputting them to the display unit 42.

[0080] Subsequently, the control unit 43 determines whether the worker who has reviewed the confirmation image G has determined that there are no problems with the target path A (step S90). That is, the worker who has reviewed the confirmation image G performs an operation to input the confirmation result through the input unit 41 (see Figure 2). Based on the operation of the input unit 41 by the worker, the control unit 43 determines whether the worker has determined that there are no problems with the target path A.

[0081] If the above step S90 is determined to be YES and it is confirmed that there are no problems with the target route A, the control unit 43 continues the automatic driving (S100) and terminates this routine.

[0082] On the other hand, if the result in step S90 is NO and it is confirmed that there is a problem with the target path A, the control unit 43 stops the automatic operation (S110) and terminates this routine. Examples of cases where there is a problem with the target path A are as follows: The first example is when the tip attachment 15c does not make contact with the workpiece O. The second example is when the position of the tip attachment 15c is too low and presses the container 27 (for example, the container floor 27a) with excessive pressure through the workpiece O. The third example is when the position of the tip attachment 15c goes too far to the back X1 and interferes with the container 27 (for example, the container front 27d). If the operator determines that any of these situations may occur, in step S90, the operator inputs an operation to the input unit 41 indicating that there is a problem with the target path A.

[0083] Although not shown in the diagram, the image generation system 1 may be configured to allow correction of the target path A (see Figure 1) via the input unit 41 (see Figure 2). Corrections to the target path A may also be reflected in the confirmation image G (see Figures 3-5). Examples of corrections to the target path A are as follows: The first example is a correction to move the position of the target path A upwards because the position of the tip attachment 15c is too low (pressing the container 27 too hard). The second example is a correction to move the position of the target path A downwards because the position of the tip attachment 15c is too low (not hitting the workpiece O). The confirmation image G may be analyzed before it is displayed, or the correction of the target path A may be performed automatically as a result of the analysis of the confirmation image G.

[0084] (Summary) The characteristic configuration and effects of the image generation system 1 described above can be summarized as follows.

[0085] [Configuration 1] The image generation system 1 is used in a work machine 10 that can perform work on a work object O contained in a container 27 by automatic operation. The image generation system 1 has a detection unit 31 and a control unit 43. The detection unit 31 detects the position information of the container 27 and the work object O contained in the container 27. The control unit 43 determines the target path A for the work inside the container 27 and, based on the determined target path A and the position information detected by the detection unit 31, generates a confirmation image G which is an image that simultaneously displays multiple display objects including the container 27 and / or the work object O and the target path A.

[0086] With the above configuration 1, when the work machine 10 performs work along the target path A by automatic operation, it is possible to easily confirm through the confirmation image G whether or not the work affects the container 27.

[0087] [Configuration 2] The work machine 10 has a tip attachment 15c (for example, a bucket). The work is a leveling operation in which the work object O loaded into the container 27 is leveled with the tip attachment 15c.

[0088] With the above configuration 2, it is possible to easily confirm through the confirmation image G whether or not the leveling work performed by the automated operation affects the container 27.

[0089] [Configuration 3] The control unit 43 determines the trajectory range, which is the path of the tip attachment 15c when it moves, based on the dimensions of the tip attachment 15c (e.g., a bucket) and the target path A, and includes an image showing the determined trajectory range (e.g., the target path top surface image GaA shown in Figure 3) in the confirmation image G as an image showing the target path A.

[0090] With the above configuration 3, it is possible to confirm the range in which the tip attachment 15c (e.g., bucket) operates with respect to the container 27 and / or the workpiece O.

[0091] [Configuration 4] The control unit 43 includes in the confirmation image G an image showing either or both of the range of the tip attachment 15c (e.g., bucket) at the starting position A1 of the target path A and the range of the tip attachment 15c (e.g., bucket) at the ending position A2 of the target path A.

[0092] With the above configuration 4, the range of the tip attachment 15c (e.g., bucket) at the starting position A1 and / or ending position A2 of the target path A can be confirmed with respect to the container 27 and / or the work object O.

[0093] [Configuration 5] The control unit 43 includes an image showing the direction from the starting position A1 to the ending position A2 (for example, the target path side view image GbA shown in Figure 3) in the confirmation image G as an image showing the target path A.

[0094] With the above configuration 5, the direction in which the tip attachment 15c (e.g., bucket) operates can be confirmed with respect to the container 27 and / or the workpiece O.

[0095] [Configuration 6] The confirmation image G includes multiple view images GE that display the above-mentioned display target from multiple different directions.

[0096] With the above configuration 6, the target path A to the container 27 and / or the workpiece O can be confirmed from multiple directions.

[0097] [Configuration 7] The confirmation image G includes a first view image GEa, which is an image of the display target viewed from above, and a second view image GEb, which is an image of the display target viewed from the side.

[0098] With the above configuration 7, the target path A to the container 27 and / or the workpiece O can be confirmed from above and from the side through the first view image GEa and the second view image GEb.

[0099] [Configuration 8] The detection unit 31 includes an imaging device 313 that acquires three-dimensional information of the container 27 and the work object O. The control unit 43 generates a confirmation image G that includes either or both of the following: a point cloud image (container point cloud image) obtained by imaging the point cloud data from the three-dimensional information of the container 27 acquired by the imaging device 313, and a modeled image (container modeled image) obtained by imaging a model of the container 27 generated from the point cloud data. The control unit 43 also generates a confirmation image G that includes either or both of the following: a point cloud image obtained by imaging the point cloud data from the three-dimensional information of the work object O acquired by the imaging device 313 (work object point cloud image), and a modeled image (work object modeled image) obtained by imaging a model of the work object O generated from the point cloud data.

[0100] In the above configuration 8, when displaying a point cloud image based on the three-dimensional information of the container 27 and the workpiece O, the state of the container 27 and the workpiece O can be accurately grasped through the confirmation image G. Furthermore, when displaying a modeled image of the container 27 and the workpiece O, the container 27 and the workpiece O can be displayed in a manner that is easy to see.

[0101] [Configuration 9] The image generation system 1 further includes a display unit 42 that displays a confirmation image G.

[0102] With the above configuration 9, the worker can easily confirm the target path A to the container 27 and / or the work object O through the display unit 42 that displays the confirmation image G.

[0103] (Modifications) The above embodiment may be modified in various ways. For example, the number of components of the above embodiment (such as each part of the confirmation image G) (including modified versions) may be changed, and some components may not be provided. For example, modified versions of the above embodiment may be combined in various ways. For example, the arrangement of components may be changed. For example, the inclusion relationships of components may be changed in various ways. For example, something described as a subordinate component included in a higher-level component may not be included in this higher-level component, but may be included in other components. For example, something described as multiple different members or parts may be treated as a single member or part. For example, something described as a single member or part may be divided and provided as multiple different members or parts. For example, elements described as components of different screens may be displayed on the same screen. For example, each component may have only a part of each feature (function, arrangement, shape, operation, etc.).

[0104] In the above embodiment, an image generation system 1 for an automated work machine 10 was described, but the present invention may also include an invention of a method or program for causing a computer to perform each of the above operations (display, selection, setting, etc.). The functions of the image generation system 1 that realize each of the above operations may correspond to "steps" in the method and program. For example, the image generation function of the control unit 43 can be rephrased as an "image generation step" in the method and program.

Claims

1. An image generation system for a work machine capable of performing work on a work object contained in a container by automatic operation, comprising: a detection unit that detects positional information of the container and the work object contained in the container; and a control unit that determines a target path for the work within the container and generates a confirmation image which is an image that simultaneously displays a plurality of display objects including the container and / or the work object and the target path, based on the determined target path and the positional information detected by the detection unit.

2. An image generation system according to claim 1, wherein the work machine has a bucket, and the work is a leveling work in which the work object loaded into the container is leveled with the bucket.

3. An image generation system according to claim 2, wherein the control unit determines a trajectory range which is the trajectory of the bucket when the bucket moves, based on the dimensions of the bucket and the target path, and includes an image showing the determined trajectory range in the confirmation image as an image showing the target path.

4. An image generation system according to claim 3, wherein the control unit includes in the confirmation image an image showing either or both of the range of the bucket at the starting position of the target path and the range of the bucket at the ending position of the target path.

5. An image generation system according to claim 4, wherein the control unit includes in the confirmation image an image indicating the direction from the starting position to the ending position as an image indicating the target path.

6. An image generation system according to claim 1, wherein the confirmation image includes a plurality of view images that display the display target from a plurality of different directions.

7. An image generation system according to claim 6, wherein the confirmation image includes a first view image which is an image of the display object viewed from above, and a second view image which is an image of the display object viewed from the side.

8. An image generation system according to claim 1, wherein the detection unit includes an imaging device that acquires three-dimensional information of the container and the work object, and the control unit generates a confirmation image that includes either or both a container point cloud image obtained by imaging the point cloud data and a container model image obtained by imaging a model of the container generated from the point cloud data, based on point cloud data generated from the three-dimensional information of the container acquired by the imaging device, and generates a confirmation image that includes either or both a work object point cloud image obtained by imaging the point cloud data and a work object model image obtained by imaging a model of the work object generated from the point cloud data, based on point cloud data generated from the three-dimensional information of the work object acquired by the imaging device.

9. An image generation system according to any one of claims 1 to 8, further comprising a display unit for displaying the confirmation image.

Images