Simulator device, simulation system, and simulator control method

Abstract

The present invention provides technology for a work machine simulator including a virtual monitor. One aspect of the present disclosure relates to a simulator device having a controller that generates a three-dimensional model of a work machine in a virtual space, moves the three-dimensional model, and displays the movement of the three-dimensional model on a virtual monitor.

Description

Simulator apparatus, simulation system, and simulator control method

[0001] The present disclosure relates to a simulator apparatus, a simulation system, and a simulator control method.

[0002] In order to perform construction work by operating working machines such as hydraulic excavators and bulldozers, practice, training, experience, or the like for mastering the operation of the working machine is required. For such operation training of working machines, not only actual machines but also simulators of working machines can be used.

[0003] With the recent advancement of information and communication technology, various electronic devices are installed in working machines. For example, a working machine is equipped with a monitor that supports touch operation. An operator can operate the working machine by operating a screen on the monitor.

[0004] Japanese Unexamined Patent Application Publication No. 2022-074843

[0005] When creating a working machine provided with a monitor in a virtual space, it is desired that not only three-dimensional data of the working machine but also a virtual monitor operable by an operator be displayed in the virtual space.

[0006] An object of the present disclosure is to provide a technology for a simulator for a working machine provided with a virtual monitor.

[0007] One aspect of the present disclosure relates to a simulator apparatus including a controller that generates a three-dimensional model of a working machine in a virtual space, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

[0008] Another aspect of the present disclosure relates to a simulation system including a simulator apparatus and an operator interface device, wherein the simulator apparatus includes a controller that generates a three-dimensional model of a working machine in a virtual space using the operator interface device, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

[0009] Another aspect of this disclosure relates to a simulator control method in which a computer generates a three-dimensional model of a work machine in a virtual space, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

[0010] According to this disclosure, we can provide technology for a simulator of a work machine equipped with a virtual monitor.

[0011] Figure 1 is a schematic diagram showing a simulation system according to the first embodiment of this disclosure. Figure 2 is a diagram showing a simulator image of a work machine according to the first embodiment of this disclosure. Figure 3 is a block diagram showing the hardware configuration of a simulator device according to the first embodiment of this disclosure. Figure 4 is a flowchart showing the simulator control process according to the first embodiment of this disclosure. Figure 5 is an image diagram showing the screen of a virtual monitor according to the first embodiment of this disclosure. Figure 6 is an image diagram showing the screen of a virtual monitor according to the first embodiment of this disclosure. Figure 7 is an image diagram showing the screen of a virtual monitor according to the first embodiment of this disclosure. Figure 8 is an image diagram showing the screen of a virtual monitor according to the first embodiment of this disclosure. Figure 9 is a diagram showing a simulator image of a work machine according to the first embodiment of this disclosure. Figure 10 is a flowchart showing the simulator control process according to the second embodiment of this disclosure. Figures 11A and 11B are image diagrams showing a real image and a virtual image according to the second embodiment of this disclosure. Figures 12A and 12B are image diagrams showing a simulator image due to left rotation according to the second embodiment of this disclosure. Figures 13A and 13B are illustrative diagrams showing simulator images obtained by right-hand rotation according to a second embodiment of the present disclosure.

[0012] Embodiments of this disclosure will be described below with reference to the drawings.

[0013] The following embodiments disclose a simulator control technology for work machines that utilizes a virtual space.

[0014] [Outline] The simulation system 20 comprises a simulator device 100 and an operator interface device. The simulator device 100 according to the following embodiment displays the view from the driver's seat of the work machine 10 in a virtual space and simulates the operating status of the work machine 10 operated by the operator in the virtual space. As shown in Figure 1, the work machine 10 is an excavator such as a shovel. For example, the work machine 10 is not limited to a shovel, but may be a wheel loader, bulldozer, etc. The operator wears VR (Virtual Reality) goggles 60. The simulator device 100 displays the view from the driver's seat of the work machine 10 as a simulator image VR1 in the virtual space within the VR goggles 60.

[0015] The VR goggles 60 are wirelessly connected to the simulator device 100. The simulator device 100 is implemented by a computing device such as a server or a personal computer (PC). The simulator image VR1 generated by the simulator device 100 is displayed on the display inside the VR goggles 60. In the following embodiment, the VR goggles 60 are described as a computing device independent of the simulator device 100. The simulator device 100 is not necessarily limited to being implemented as a computing device independent of the VR goggles 60. The simulator device 100 may be provided within the VR goggles 60.

[0016] The operator puts on VR goggles 60 on their head. While viewing the simulator image VR1 displayed inside the VR goggles 60, the operator operates the control levers 50 and pedals 40 installed in the simulator facility. The operator's operations cause the work machine 10 displayed in the simulator image VR1 to operate.

[0017] The cockpit of the actual work machine 10 is equipped with a control monitor for setting and controlling the work machine 10, and for displaying the operating status of the work machine 10. The operator can set and control the work machine 10 and check its operating status not only by using the control levers and pedals in the cockpit, but also by touching the screen of the control monitor.

[0018] The simulator image VR1 is configured to include a virtual monitor 80 in which a control monitor is placed in a virtual space. The operator touches the virtual monitor 80 displayed in the simulator image VR1 with their fingers detected by hand tracking technology using distance measuring sensors such as a camera mounted on the VR goggles 60, similar to how they would operate the control monitor of a real machine. By touching the virtual monitor 80, the operator can set and control the 3D model of the work machine 10 in the virtual space and check its operating status. However, the detection of operator operations according to this disclosure is not necessarily limited to hand tracking technology. For example, operator operations may be detected by a glove-type VR controller worn by the operator.

[0019] In the embodiments described later, the simulator device 100 generates a three-dimensional model of the work machine 10 operated by the operator in a virtual space. The simulator device 100 projects the generated three-dimensional model as a simulator image VR1 into the VR goggles 60 worn by the operator. Typically, such a three-dimensional model is created in advance by a CAD (Computer-Aided Design) engineer, designer, etc., and stored in the simulator device 100.

[0020] For example, if the work machine 10 is an excavator, the simulator image VR1 may be an image like the one shown in Figure 2. As shown in the figure, a virtual monitor 80 corresponding to the control monitor mounted on the actual work machine 10 is displayed in the simulator image VR1. In addition, parts of the excavator, such as the left and right operating levers 90, the left and right pedals 91, the attachment 92, the arm and boom 93, are also displayed in the simulator image VR1.

[0021] The view of the construction site and other areas from the cockpit is superimposed onto the simulator image VR1. Such a 3D model of the construction site is created in advance by a CAD engineer, designer, etc., and stored in the simulator device 100. For example, the 3D model of the construction site may be design data representing the target shape of the construction site. For example, when the operator operates the 3D model of the work machine 10 by operating the joystick-type control lever 50, the view from the cockpit changes.

[0022] The simulator image VR1 is a composite image generated by superimposing a virtual image L1, which projects the cockpit of the 3D model of the work machine 10 and the view from that cockpit, with a background image U1. The background image U1 is the portion of the simulator image VR1 that is not the virtual image L1. The background image U1 is, for example, a real image of the real space in front of the operator. For example, the background image U1 may be an image captured by a camera installed near the operator. Alternatively, the background image U1 may be an image captured by VR goggles 60 worn by the operator. The background image U1 is not limited to these and may be an image of other real spaces. Furthermore, the background image U1 may be a monochrome image such as white or black, or an artificial image generated by a generation AI.

[0023] In the illustrated simulator image VR1, the background image U1 is positioned above in the vertical direction, and the virtual image L1 is positioned below. The virtual image L1 changes in accordance with the operator's operation of the 3D model of the work machine 10, while the background image U1 may be a video image captured in conjunction with the operator's head position or posture in real space. That is, the simulator device 100 may display a real image within the simulator image without being linked to the virtual image L1 which is changed in accordance with the operator's operation. Specifically, a video image from a camera equipped in the VR goggles 60 worn by the operator may be used as the background image U1. Since the positional relationship between the camera viewpoint, which moves in conjunction with the position and direction of the camera equipped in the VR goggles 60, and the position of the operator's eyes is maintained, a background image U1 that does not feel unnatural to the operator can be displayed. Since the position and direction of the camera of the VR goggles 60 change in conjunction with the operator's head position and posture, the operator's movement and the change in the background image U1 are linked, so a background image U1 that does not feel unnatural to the operator can be displayed.

[0024] Note that the simulator image VR1 is not limited to the arrangement of the background image U1 and virtual image L1 as shown in the figure. For example, the simulator image VR1 may be divided into three vertical sections, with a virtual image L1 representing the frame of the work machine 10 and the ground displayed in the upper area of ​​the simulator image VR1, a background image displayed in the middle area, and a virtual image L1 representing the work machine 10 displayed in the lower area.

[0025] In this way, by including the virtual image L1 and the background image U1 in the simulator image VR1, the possibility of motion sickness in the operator can be reduced.

[0026] When an operator performs an operation on the virtual monitor 80 installed on the 3D model of the work machine 10, the simulator device 100 operates the 3D model of the work machine 10 based on that operation on the virtual monitor 80. The simulator device 100 controls the simulator image VR1 displayed on the virtual monitor 80 according to the operating status of the 3D model of the work machine 10. In other words, the simulator device 100 controls the simulator image VR1 displayed on the virtual monitor 80 so that it displays an image similar to the image displayed on the control monitor installed on the actual work machine 10, according to the operating status of the 3D model of the work machine 10.

[0027] By displaying a virtual monitor 80 on the 3D model that is similar to the control monitor installed on the actual work machine 10, the operator can operate the 3D model in the same way as the actual machine.

[0028] The simulator device 100 may be implemented using computing devices such as a server, personal computer (PC), smartphone, or tablet. For example, it may have a hardware configuration as shown in Figure 3. That is, the simulator device 100 includes a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, all interconnected via bus B.

[0029] The programs or instructions that implement various functions and processes in the simulator device 100 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a USB (Universal Serial Bus) memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may also be downloaded from any external device via a network or the like.

[0030] The storage device 102 is implemented by a hard disk drive or the like, and stores files, data, etc., used to execute the installed program or instruction, along with the program or instruction itself.

[0031] The memory device 103 is implemented using random access memory, static memory, etc., and when a program or instruction is activated, it reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitor storage medium.

[0032] The processor 104 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc., which may consist of one or more processor cores. The processor 104 executes various functions and processes of the simulator device 100 according to programs, instructions, data such as parameters necessary to execute the programs or instructions stored in the memory device 103.

[0033] The user interface (UI) device 105 provides an interface between the user and the simulator device 100. The user interface device 105 may consist of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, and headset, and input / output devices such as a touch panel. For example, the user may operate the simulator device 100 by using a keyboard, mouse, etc., to operate a GUI (Graphical User Interface) displayed on a display or touch panel.

[0034] The communication device 106 is implemented by various communication circuits that perform wired and / or wireless communication processing with external devices, the Internet, LAN (Local Area Network), cellular networks, and other communication networks.

[0035] However, the hardware configuration described above is merely an example, and the simulator device 100 according to this disclosure may be implemented using any other suitable hardware configuration.

[0036] The pedal 40, operating lever 50, and VR goggles 60 shown in Figure 1 can be collectively referred to as an operator interface device that transmits communication between the simulator device 100 and the operator. The operator interface device according to this disclosure is not necessarily limited to the pedal 40, operating lever 50, and VR goggles 60. Other devices depending on the various forms of the work machine 10 may be used as the operator interface device. For example, if the operator wears a glove-type VR controller, the glove-type VR controller may be included in the operator interface device. The simulation system 20 may consist of the simulator device 100 and the operator interface device.

[0037] [Simulator Control Processing] Next, the simulator control processing according to the first embodiment of the present disclosure will be described. This simulator control processing is performed by a simulator device 100 implemented by a server, PC, etc. A controller 110 implemented by the processor 104 of the simulator device 100 executes the simulator control processing according to programs, instructions and / or data stored in a non-transitor computer-readable storage medium (non-transitor computer-readable storage medium) of a storage device 102 and a memory device 103, etc. (hereinafter collectively referred to as memory 120). Figure 4 is a flowchart showing the simulator control processing according to one embodiment of the present disclosure.

[0038] In step S101, the controller 110 generates a three-dimensional model of the work machine 10 in a virtual space. The controller 110 places a virtual monitor 80 on the three-dimensional model. The controller 110 extracts the three-dimensional model of the work machine 10 from memory 120 and also obtains a three-dimensional model representing the virtual work environment, such as a construction site, from memory 120.

[0039] If multiple 3D models of the work machine 10 and multiple 3D models representing the virtual work environment are stored in memory 120, the operator can specify one of the 3D models. The controller 110 may project a composite image of the 3D model of the work machine 10 specified by the operator and the 3D model of the virtual work environment onto the VR goggles 60 as simulator image VR1.

[0040] When the operator puts on the VR goggles 60, the simulator device 100 may prompt the operator to select a 3D model of the work machine 10 and a 3D model of the virtual work environment.

[0041] In step S102, the controller 110 receives operator input to a virtual monitor 80 installed on the work machine 10. For example, the actual work machine 10 is provided by the control monitor. The operator uses the control monitor to set or control the work machine 10 and to check the operating status of the work machine 10 (e.g., direction of movement, speed of movement, altitude, etc.). The controller 110 provides a virtual monitor 80 corresponding to the control monitor in a three-dimensional model of the work machine 10.

[0042] The virtual monitor 80 displays a screen similar to the control monitor of the actual machine. The controller 110 performs screen transitions in the same way that an operator would operate the control monitor of the actual machine. For example, when an operator starts operating the 3D model of the work machine 10, the controller 110 may display a predetermined main menu screen for setting the 3D model of the work machine 10 on the virtual monitor 80.

[0043] The operator selects an item from the main menu screen by touching the screen of the virtual monitor 80 in the virtual space. The controller 110 detects the operator's selection operation and accepts the selected item.

[0044] As shown in Fig. 5, an operator selects the item "Mode Setting" from the main menu screen displayed on the virtual monitor 80. When the controller 110 detects a touch operation by the operator's fingers recognized through hand tracking, it detects that the item "Mode Setting" has been selected by the operator.

[0045] Mode setting is an item for setting the operation mode of the work machine 10. Examples include a fully manual mode for fully manually operating the three-dimensional model of the work machine 10, an automatic movement mode for automatically moving the three-dimensional model of the work machine 10 to a predetermined position on a construction site, and an automatic excavation mode for causing the three-dimensional model of the work machine 10 to automatically excavate the ground at a predetermined position. In this manner, the controller 110 may accept automatic control instructions related to construction operations by the work machine 10 on the virtual monitor 80. For example, such automatic control instructions include automatic movement mode, automatic excavation mode, or automatic swing mode for driving an arm and a boom 93 while swinging a swing structure of the work machine 10, and may be semi-automatic control.

[0046] The controller 110 may change the screen displayed in response to an operation on the virtual monitor 80. That is, when the operator selects the item "Mode Setting" on the virtual monitor 80 shown in Fig. 5, the controller 110 transitions the screen displayed on the virtual monitor 80 from the main menu screen to the mode setting screen. For example, the screen may transition to and be displayed as a mode setting screen as shown in Fig. 6.

[0047] An operation on the virtual monitor 80 may be based on position information and time information of the operator's finger operations recognized by hand tracking. The controller 110 can specify at which position and for how long each of the operator's left and right fingers stayed based on the position information and time information of the fingers detected by hand tracking. This allows the controller 110 to determine an operation performed by the operator on the three-dimensional model of the work machine 10 in the virtual space.

[0048] The controller 110 may detect an operator's touch operation on a virtual monitor 80 in a virtual space based on finger position information and time information detected by hand tracking. The operation on the virtual monitor 80 may include a touch operation on the virtual monitor 80 performed by the operator. For example, when it is detected that the operator's index finger stays on an item displayed on the screen of the virtual monitor 80 for a predetermined period of time or longer, the controller 110 may determine that the item has been selected.

[0049] The touch operation may include various operations such as swipe, long press, and pinch used in smartphones, tablets, and the like. The controller 110 may determine what type of operation the operator has performed according to the detected operation content.

[0050] When the controller 110 detects that the operator has touched the virtual monitor 80 by hand tracking, the controller 110 detects the touched position on the virtual monitor 80. The controller 110 performs coordinate conversion on the touched position on the virtual monitor 80 to the monitor position of a simulation PC in the real world. The controller 110 changes a monitor image in accordance with the coordinate-converted monitor position. The controller 110 copies a monitor screen displayed on the monitor of the simulation PC in the real world, pastes the copied monitor screen as a texture onto the virtual monitor 80 in the virtual space, and displays the pasted monitor screen. The display timing may be any timing or real time. The texture may be pasted onto the virtual monitor 80 from moment to moment. The image displayed on the virtual monitor 80 may be a bird's-eye view obtained by viewing the periphery of a three-dimensional model from above.

[0051] In step S103, the controller 110 operates the work machine 10 in the virtual space based on operations on the virtual monitor 80. The controller 110 operates a 3D model of the work machine 10 in the virtual space based on operations on the virtual monitor 80. The operations on the virtual monitor 80 are based on positional and temporal information of operations performed by the operator's fingers, which are recognized by hand tracking. The controller 110 operates the 3D model of the work machine 10 according to the setting information received on the virtual monitor 80 and projects a simulator image VR1 of the operating 3D model onto the VR goggles 60.

[0052] When "Automatic Movement Mode" is selected in the mode setting screen as shown in Figure 6, the controller 110 moves the 3D model of the work machine 10 to the set target position using "Automatic Movement Mode".

[0053] In step S104, the controller 110 displays the operation of the 3D model on the virtual monitor 80. The controller 110 controls the images displayed on the virtual monitor 80 according to the operating status of the 3D model. It controls the images displayed in the virtual space according to the operating status of the work machine 10. The virtual monitor 80 displayed in the virtual space may display the operating status of the work machine 10 that has been operated in response to the operator's operation on the virtual monitor 80. For example, the operating status may be the direction of movement, speed of movement, and altitude of the 3D model of the work machine 10. For example, the operating status may indicate the current position of the 3D model of the work machine 10 in the virtual space, the movement path that the 3D model of the work machine 10 should take, and the surrounding conditions of the 3D model of the work machine 10 in the work environment.

[0054] As shown in Figure 7, the virtual monitor 80 displays the current position and movement path of the 3D model of the work machine 10 at the top of the screen, and the surrounding conditions of the 3D model of the work machine 10 at the bottom of the screen. The virtual monitor 80 dynamically changes the current position, movement path, and surrounding conditions according to the movement of the 3D model of the work machine 10.

[0055] When the work machine 10 is operating using the automatic control function, the controller 110 may display operations or displays related to the automatic control on the virtual monitor 80. The automatic control may include, for example, machine control, virtual wall, and human detection. The controller 110 may also display other work machines, people, virtual walls, teaching points, alignment lines, design planes, etc., on the virtual monitor 80. The automatic control function may include not only fully automatic operation that does not require operator intervention, but also semi-automatic / operation assist (e.g., leveling assist and stop control) that partially involves operator intervention, and excavation assist.

[0056] When the work machine is operating using the automatic control function, the controller 110 may display one or both of the alignment lines for travel and / or teaching points for operation on the virtual monitor 80 as operations or displays related to automatic control. When the work machine 10 is operating using the automatic control function, the controller 110 may display the alignment lines for travel and / or teaching points for operation on the virtual monitor 80. As shown in Figure 8, when the automatic movement mode is set, the controller 110 may display the current position of the 3D model in the work environment and the movement path of the automatic movement shown by the dashed alignment lines on the virtual monitor 80. The controller 110 may also display teaching points for instructing the operator on operation on the virtual monitor 80. In the illustrated example, the controller 110 may display "Move straight for 15m from the current position, then turn 45 degrees to the right" as a teaching point on the virtual monitor 80.

[0057] In automatic rotation control, the rotating body is automatically rotated to move the work machine 10 to a target position. The operator needs to pre-set the target position by operating the virtual monitor 80, and the controller 110 may display teaching content for this purpose on the virtual monitor 80. For example, the operator sets the excavation preparation position (return point), interference avoidance point, and loading position (earth discharge point on the truck or other loading platform) on the virtual monitor 80. The controller 110 may display these three set points as teaching points on the virtual monitor 80 or in the virtual space. The controller 110 may rotate the rotating body in the virtual space based on a reference position and / or reference plane and drive the work machine 10. This makes it possible to know the trajectory of the automatic rotation control in advance.

[0058] When the automatic movement mode is selected in the mode settings, the controller 110 moves the 3D model of the work machine 10 to the set target position using the automatic movement mode. The controller 110 changes the field of view from the 3D model of the work machine 10 as it moves in the virtual space, and projects the simulator image VR1, which includes the changing field of view, onto the VR goggles 60.

[0059] Operations on the virtual monitor 80 may involve setting one or more reference positions and reference planes. The controller 110 may display content on the virtual monitor 80 indicating the reference positions and reference planes. For example, if the work machine 10 is equipped with a slewing body and a work machine, such as an excavator, the controller 110 may rotate the slewing body and drive the work machine based on one or more reference positions. When performing machine control leveling assist in the simulator, the controller 110 may drive the work machine 10 based on reference planes. The reference positions and / or reference planes may be set by the operator operating the virtual monitor 80, or they may be pre-set.

[0060] For example, if the turning point of the work machine 10 is set as the reference position in automatic movement mode, the controller 110 may display the movement path including the turning point in the simulator image. As shown in Figure 9, the controller 110 may display the reference position P1 in the simulator image and display the simulator image so that the rotating body of the 3D model rotates at the reference position P1 and the work machine operates. The controller 110 may also display icons indicating the reference position or content indicating the reference plane on the virtual monitor 80.

[0061] The controller 110 may output sound in conjunction with the image displayed on the virtual monitor 80. For example, as described above, when a reference position and / or reference plane is set, the controller 110 may provide the operator with guidance information (e.g., alignment lines, teaching points, etc.) to perform a predetermined task based on the setting information. The guidance information may also be notified to the operator as voice guidance.

[0062] If the operator does not perform the operation in accordance with the guidance information, the controller 110 may provide the operator with voice guidance notifying them that the operation was inappropriate.

[0063] According to the simulator control process described above, if the actual machine 10 is equipped with a control monitor for setting operation and checking operating status, the screen displayed on the virtual monitor 80 corresponding to the control monitor is also displayed on the simulator image VR1 of the 3D model of the machine 10. The screen displayed on the virtual monitor 80 can be controlled in the same way as the screen displayed on the control monitor of the actual machine.

[0064] Next, a simulator control process according to a second embodiment of the present disclosure will be described. This simulator control process is executed by a simulator device 100 implemented by a server, PC, etc. More specifically, a controller 110 implemented by a processor 104 of the simulator device 100 executes the simulator control process according to programs, instructions and / or data stored in a non-transitor computer-readable storage medium (hereinafter collectively referred to as memory 120) of a storage device 102 and a memory device 103, etc. Figure 10 is a flowchart showing a simulator control process according to one embodiment of the present disclosure.

[0065] In step S101, the controller 110 acquires a background image U1 and a virtual image L1 of a three-dimensional model of the work machine 10 operated by the operator. The background image U1 may be an image captured in real space by a camera mounted on VR goggles 60 worn on the operator's head. Specifically, the background image U1 may be a video image captured in conjunction with the position or posture of the operator's head in real space. As shown in Figure 11A, it may be a video image captured in front of the operator performing a simulation using a three-dimensional model of the work machine 10.

[0066] On the other hand, the virtual image L1 may be generated based on a three-dimensional model of the work machine 10 stored in memory 120, as shown in Figure 11B. The controller 110 extracts the three-dimensional model from memory 120 and also obtains a three-dimensional model representing the virtual work environment, such as a construction site, from memory 120. The three-dimensional model representing the virtual work environment is also the ground in the virtual space. The virtual image L1 includes the three-dimensional model of the work machine 10 and the ground in the virtual space representing the virtual work environment.

[0067] If multiple 3D models of the work machine 10 and multiple 3D models representing the virtual work environment are stored in the memory 120, the administrator of the simulator device 100 can specify one of the 3D models. The controller 110 may project a composite image of the 3D model of the work machine 10 and the 3D model of the virtual work environment specified by the administrator onto the VR goggles 60 as simulator image VR1.

[0068] For example, when an operator puts on VR goggles 60, the simulator device 100 requests the operator to select a 3D model of the work machine 10 and a 3D model of the virtual work environment. The controller 110 may project a composite image of the 3D model of the work machine 10 and the 3D model of the virtual work environment selected by the operator onto the VR goggles 60 as simulator image VR1.

[0069] In step S102, the controller 110 generates a composite image based on the background image U1 and the virtual image L1. Specifically, the controller 110 may combine the background image U1 shown in Figure 11A and the virtual image L1 shown in Figure 11B to generate a composite image as shown in Figure 2. As shown in Figure 2, the controller 110 generates a simulator image VR1 in which the background image U1 and the virtual image L1 are arranged vertically. The controller 110 may combine the background image U1 and the virtual image L1 in a vertical ratio of 1:1. The disclosure is not limited thereto, and the background image U1 and the virtual image L1 may be arranged in other ways. For example, the controller 110 may variably arrange the display areas of the background image U1 and the virtual image L1. The controller 110 may divide the display area of ​​the composite image into three vertical sections and generate it. The upper section of the three divided display areas may display a virtual image L1 representing the frame of the work machine 10. A background image U1 may be displayed in the middle of the three divided display area. A virtual image L1 representing the cockpit of the work machine 10 may be displayed in the lower of the three divided display area. The work machine 10 is equipped with a driver's cab including a front window and side windows, and a composite image may be displayed on both the front window and the side windows.

[0070] The composite image may be generated by superimposing the background image U1 and the virtual image L1 onto the front window of the work machine 10, and superimposing one or both of the background image U1 and the virtual image L1 onto the side window of the work machine 10. For example, the controller 110 may generate the composite image by superimposing the background image U1 and the virtual image L1 onto the front window of the 3D model of the work machine 10, and superimposing the background image U1 onto the side window. Alternatively, the controller 110 may generate the composite image by superimposing the background image U1 and the virtual image L1 onto the front window of the virtual machine of the work machine 10, and superimposing the virtual image L1 onto the side window.

[0071] In step S103, the controller 110 controls the composite image displayed in the virtual space in response to the operator's actions. Specifically, the 3D model of the work machine 10 moves in response to the operator's actions, and the virtual image L1 displayed in the virtual space changes in response to these actions. The controller 110 may also change the virtual image L1 in response to the operator's actions. In other words, the controller 110 controls the composite image displayed in the virtual space in response to the movement of the 3D model of the work machine 10. For example, if the work machine 10 is equipped with an upper rotating body, a lower traveling body, and a work machine like an excavator, when the operator rotates the 3D model of the work machine 10, the controller 110 changes the virtual image L1 superimposed on the simulator image in accordance with the rotation, similar to the operator's view when the actual work machine 10 is rotating. The rotation operation of the work machine 10 may be based on either an operation on the lower traveling body of the work machine 10, an operation on the upper rotating body of the work machine 10, or a combination thereof. For example, the turning motion may be performed by rotating the left and right travel motors of the lower travel body in opposite directions. Alternatively, the turning motion may be performed by rotating the upper slewing body using the slewing motor of the upper slewing body with respect to the lower travel body. The controller 110 displays the background image U1 in the composite image without being linked to the virtual image L1 which is changed in accordance with the turning operation of the work machine 10.

[0072] When the operator rotates the 3D model of the work machine 10 to the left, the controller 110 changes the simulator image VR2, as shown in Figure 12A, to the simulator image VR3, as shown in Figure 12B. That is, as shown in the figure, of the background image U2 and virtual image L2 displayed in the simulator image VR2 before the left rotation, in the simulator image VR3 after the left rotation, the field of view of the background image U3 remains unchanged from the background image U2, while the field of view of the virtual image L3 changes from that of the virtual image L2 after the left rotation.

[0073] When the operator rotates the 3D model of the work machine 10 to the right, the controller 110 changes the simulator image VR2, as shown in Figure 13A, to the simulator image VR4, as shown in Figure 13B. That is, as shown in the figure, of the background image U2 and virtual image L2 displayed in the simulator image VR2 before the right rotation, in the simulator image VR4 after the right rotation, the field of view of the background image U4 remains unchanged from the background image U2, while the field of view of the virtual image L4 changes from the virtual image L2 to one that has been rotated to the right.

[0074] The controller 110 displays background images U3 and U4 in the simulator images VR3 and VR4 without being linked to the virtual images L3 and L4, which change in response to the operator's actions. In other words, background images U3 and U4 are displayed independently of the dynamic changes in virtual images L3 and L4, which change as the field of view changes with rotation. The composite image displays background images U3 and U4 without being linked to the virtual images L3 and L4. The controller 110 displays background images U3 and U4 without being linked to the virtual images L3 and L4, which are changed in response to the rotational movement of the 3D model. As a result, the operator's perception matches the movement of background images U3 and U4, which can reduce the sense of immersion the operator gets from the simulator images VR3 and VR4 and potentially reduce motion sickness in the operator. In the illustrated example, background images U3 and U4 are the same before and after rotation, but this disclosure is not limited to this, and background images U3 and U4 may change in accordance with the movement of people or objects in the real space being captured. Background images U3 and U4 are video images captured in conjunction with the camera's position and orientation in real space.

[0075] Furthermore, the controller 110 may generate a composite image that variably includes the display areas of the background image U1 and the virtual image L1 depending on the type of object to be displayed. For example, "ground" and "working machinery" may be assumed to be objects to be displayed in the composite image, and the controller 110 may generate a composite image according to the type of object to be displayed. For example, the controller 110 may display "ground" and "working machinery" that are not work targets as virtual images L1 in the composite image, and display the other objects to be displayed as background images U1.

[0076] According to the simulator control process described above, a virtual image L1 that changes according to the operating status of the work machine 10 is projected onto the simulator image, along with a background image U1 that is displayed independently of the operating status. This reduces motion sickness caused by the rotation of the 3D model of the work machine 10.

[0077] In the embodiments described above, the simulator image VR1 was controlled based on detection of the operator's finger movements on the virtual monitor 80, but this disclosure is not limited thereto. For example, the simulator image VR1 may be controlled by detecting movements of other body parts (e.g., head, arms, legs, etc.) other than the fingers. The simulator image VR1 may also be controlled by detecting operations of the operator on virtual objects other than the virtual monitor 80 (e.g., virtual control levers, virtual pedals, virtual objects or software in the cockpit of the work machine 10, etc.), rather than being limited to touch operations on the virtual monitor 80.

[0078] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of this disclosure as described in the claims.

[0079] 10. Work machinery 20. Simulation system 40. Pedal 50. Operating lever 60. VR goggles 80. Virtual monitor

Claims

1. A simulator device having a controller that generates a three-dimensional model of a work machine in a virtual space, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

2. The simulator apparatus according to claim 1, wherein the virtual monitor is installed on the three-dimensional model.

3. The simulator apparatus according to claim 1 or claim 2, further comprising accepting operations on the virtual monitor.

4. The simulator apparatus according to claim 3, wherein operating the three-dimensional model includes operating the three-dimensional model of the work machine in the virtual space based on the operation on the virtual monitor.

5. The simulator apparatus according to claim 4, wherein displaying the operation of the three-dimensional model on the virtual monitor includes controlling the image displayed on the virtual monitor according to the operation status of the three-dimensional model.

6. The simulator apparatus according to claim 5, wherein controlling the image includes displaying operations or displays related to automatic control on the virtual monitor when the work machine is operating by an automatic control function.

7. The simulator device according to claim 6, wherein displaying operations or displays related to the automatic control on the virtual monitor includes displaying one or both of the alignment lines related to driving and teaching points related to operations on the virtual monitor.

8. The simulator device according to claim 3, wherein the operation of the virtual monitor is an operation by an operator, and is based on positional and temporal information of the operator's finger operations recognized by hand tracking.

9. The simulator apparatus according to claim 8, wherein the operation of the virtual monitor includes touch operations on the virtual monitor by the operator.

10. The simulator apparatus according to claim 5, wherein controlling the image includes changing the displayed screen in response to an operation on the virtual monitor.

11. The simulator apparatus according to claim 5, wherein the operation on the virtual monitor includes setting one or both of one or more reference positions and reference planes, and the control of the image includes displaying content on the virtual monitor that represents one or both of the one or more reference positions and reference planes.

12. The simulator apparatus according to claim 1, further comprising outputting sound in conjunction with the image displayed on the virtual monitor.

13. The simulator device according to claim 3, wherein the virtual monitor receives automatic control instructions regarding construction operations based on the three-dimensional model.

14. A simulator device according to claim 1, which generates a composite image based on a background image and a virtual image including the three-dimensional model of the work machine, and controls the composite image displayed in the virtual space according to the operation of the three-dimensional model.

15. The simulator apparatus according to claim 14, wherein the background image is a real image in real space.

16. The simulator apparatus according to claim 14, wherein the virtual image includes the ground in the virtual space.

17. The simulator device according to claim 14, wherein the background image is a video image captured in conjunction with the position and direction of the camera in real space, and the camera is mounted on the operator's head.

18. The simulator apparatus according to claim 14, wherein the composite image displays the background image without being linked to the virtual image.

19. A simulation system comprising a simulator device and an operator interface device, wherein the simulator device has a controller that generates a three-dimensional model of a work machine in a virtual space using the operator interface device, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

20. A simulator control method in which a computer generates a three-dimensional model of a work machine in a virtual space, operates the three-dimensional model, and displays the operation of the three-dimensional model on a virtual monitor.

Images