Work assistance system and work assistance method

Abstract

According to an embodiment, this work assistance system comprises: a first MR device; a second MR device that is capable of communicating with the first MR device over a network; and a subjective viewpoint camera that acquires subjective viewpoint video data by capturing a subjective viewpoint video of a first user wearing the first MR device. The first MR device comprises an eyeball imaging camera and a transmission unit. The eyeball imaging camera acquires line-of-sight coordinate data by imaging the eyeballs of the first user. The transmission unit transmits the line-of-sight coordinate data and the subjective viewpoint video data of the first user to the second MR device over the network. The second MR device comprises a reception unit, a cursor generation unit, and a display unit. The reception unit receives the line-of-sight coordinate data and the subjective viewpoint video data of the first user. The cursor generation unit generates cursor information indicating a point of interest of the first user from the received line-of-sight coordinate data. The display unit superimposes the cursor information on the received subjective viewpoint video data of the first user and displays the superimposed data in the field of view of a second user wearing the second MR device.

Description

Work Support System and Work Support Method

[0001] The embodiments relate to a work support system and a work support method.

[0002] The work support system is a system for remotely supporting on-site workers to help the work be performed smoothly. Specifically, a controller who assists the worker communicates with the worker via a network, and the two parties work collaboratively, which can achieve effects such as faster work progress, higher work safety, and cost reduction.

[0003] Conventionally, there has been known a technology in which a viewpoint image of a worker wearing an MR (Mixed Reality) device is transmitted to a controller, and the controller who has viewed the image transmits a desired action as hand model information to the worker (see Non-Patent Document 1). With this technology, by superimposing (overlaying) and displaying necessary information on the worker's MR device, the controller's judgments and information can be efficiently conveyed to the worker.

[0004] Hideaki Tanaka, Masashi Tadokoro, Haruo Oishi, & Yusuke Morita. (2024). Verification of Effectiveness of Remote Operation Support via Hand Model Transmission Using MR Device. IEICE Technical Report, vol. 124, no. 53, ICM2024-9, pp. 40-45, May 2024.

[0005] In existing technologies, there has been no way for a worker to notify the controller that "the worker has correctly recognized the controller's request". For this reason, there has been a risk that work will proceed while there is a misunderstanding between the two parties.

[0006] The present invention has been made in view of the above circumstances, and an object thereof is to provide a technology capable of accurately feeding back a worker's judgment to a controller.

[0007] A work support system according to one aspect of this invention comprises a first MR device, a second MR device capable of communicating with the first MR device via a network, and a subjective viewpoint camera that captures subjective viewpoint video of a first user wearing the first MR device and acquires subjective viewpoint video data. The first MR device comprises an eyeball imaging camera and a transmission unit. The eyeball imaging camera captures the first user's eyeballs and acquires gaze coordinate data, and the transmission unit transmits the first user's subjective viewpoint video data and gaze coordinate data to the second MR device via the network. The second MR device comprises a receiving unit, a cursor generation unit, and a display unit. The receiving unit receives the first user's subjective viewpoint video data and gaze coordinate data. The cursor generation unit generates cursor information indicating the first user's attention points from the received gaze coordinate data. The display unit superimposes the cursor information onto the received subjective viewpoint video data of the first user and displays it in the field of view of the second user wearing the second MR device.

[0008] According to this embodiment, it becomes possible to accurately feed back the worker's judgment to the supervisor.

[0009] Figure 1 is a system diagram showing an example of a work support system according to an embodiment. Figure 2 is a functional block diagram showing an example of a work support system according to the first embodiment. Figure 3 is a flowchart showing an example of a processing procedure in the controller's MR device 12. Figure 4 is a flowchart showing an example of a processing procedure in the worker's MR device 20. Figure 5 is a diagram illustrating an example of a conversation between a worker and a controller. Figure 6 is a functional block diagram showing an example of a work support system according to a second embodiment. Figure 7 is a functional block diagram showing an example of a work support system according to a third embodiment. Figure 8 is a block diagram showing an example of the hardware configuration of MR devices 12 and 20.

[0010] Figure 1 is a system diagram showing an example of a work support system according to the embodiment. The work support system is a system that creates a communication environment between workers and supervisors and supports workers on site from a remote control center with video, audio, etc. This type of technology is also known as remote OP (operation) technology, and is a technology that exchanges subjective viewpoint video and enables communication between workers and supervisors on that video.

[0011] In Figure 1, the worker, acting as the first user, wears an MR device 20. Additionally, a camera 22 is attached to the worker's helmet, for example, to capture the worker's subjective viewpoint and eye movements. The video from the worker's subjective viewpoint and the data obtained from capturing the eye movements are transmitted to the controller's MR device 12 via a network 100, such as an IP (Internet Protocol) network.

[0012] The controller, acting as a second user, wears an MR device 12. Additionally, a camera 11 is attached to the controller's headphones, for example, to capture the controller's subjective viewpoint and hands. The controller's hands are then transmitted via the network 100 to the worker's MR device 20.

[0013] In the work support system shown in Figure 1, there are situations where a supervisor, located away from the work site where the workers are, remotely performs work by monitoring the site conditions and the workers' behavior, making appropriate judgments, and issuing work instructions (controlling the work). This type of work arises when a supervisor capable of making complex judgments (which may be a person or some kind of information processing device) cannot become a worker (which may be a person or a robot) who moves to the site to perform the work. In such situations, video from the worker's subjective viewpoint may be sent to the supervisor, who then views it to obtain information from the site, make judgments, and transmit instructions to the workers. In this case, it is important that the supervisor's judgments are accurately conveyed to the workers.

[0014] Existing technologies could transmit the controller's judgment to the worker. However, they could not guarantee that the worker correctly understood the transmitted judgment. The following discloses a technology that can eliminate such concerns.

[0015] [First Embodiment] <Example of transmitting the worker's line of sight position to the controller> Figure 2 is a functional block diagram showing an example of a work support system according to the first embodiment.

[0016] The MR device 20 includes an eyeball imaging camera 22a, a subjective viewpoint camera 22b, a subjective viewpoint image gaze position calculation unit 241, a communication unit 23, a 3D hand model generation unit 242, an image superposition unit 243, and a display unit 25.

[0017] The eyeball imaging camera 22a captures the eyeballs of the worker wearing the MR device 20 and acquires gaze coordinate data. The subjective viewpoint camera 22b captures the subjective viewpoint video of the worker and acquires subjective viewpoint video data. The gaze position calculation unit 241 on the subjective viewpoint video estimates the gaze coordinates in the video by, for example, matching the relative position of the pupil in the eyeball with the subjective viewpoint video.

[0018] The communication unit 23 transmits the worker's subjective viewpoint video data and gaze coordinate data to the controller's MR device 12 via the network 100. The 3D hand model generation unit 242 generates a hand model representing the object being instructed by the controller from the controller's hand video data transmitted from the controller's MR device 12.

[0019] The video overlay unit 243 overlays a hand model onto the worker's subjective viewpoint video data and displays it on the display unit 25. As a result, the worker sees a video including the supervisor's hand model within their field of view.

[0020] The MR device 12 includes a hand-held camera 91, a hand capture unit 31, a communication unit 70, a worker's gaze position cursor generation unit 32, an image superposition unit 33, and a display unit 51.

[0021] The hand-camera 91 captures the controller's hands and acquires hand-camera video data. The hand capture unit 31 records the shape of the controller's pointing hand from the hand-camera video data. The communication unit 70 transmits the hand-camera video data, captured data, etc., to the MR device 20.

[0022] The worker gaze position cursor generation unit 32 generates cursor information indicating the worker's attention points from the worker's gaze coordinate data transmitted from the MR device 20. The video overlay unit 33 overlays the cursor onto the supervisor's subjective viewpoint video data and displays it on the display unit 51. As a result, the supervisor's field of view displays an image including the cursor indicating the worker's attention points. In the above configuration, for example, the following procedure is performed.

[0023] (1) The eye-capturing camera 22a of the worker's MR device 20 captures the worker's eyeballs. (2) The subjective viewpoint camera 22b of the worker's MR device 20 captures the worker's subjective viewpoint video. (3) The MR device 20 calculates the worker's gaze coordinates in the subjective viewpoint based on the subjective viewpoint video and the direction of the eyeballs. For example, the relative position of the pupil in the eyeball can be compared with the subjective viewpoint video to estimate the gaze coordinates in the video.

[0024] (4) The MR device 20 transmits the subjective viewpoint video and gaze coordinate information to the controller's MR device 12. (5) The MR device 12 receives the subjective viewpoint video and gaze coordinate information. (6) The MR device 12 converts the received gaze coordinates into graphic data such as a cursor. (7) The MR device 12 overlays the cursor onto the received subjective viewpoint video and generates display data.

[0025] (8) The MR device 12 displays the display data on the display unit 51. (9) The hand-held camera 91 captures the operator pointing to equipment or other items that the operator wants the operator to operate, which are visible in the operator's subjective viewpoint video. (10) The MR device 12 records the shape of the hand captured by the hand capture unit 31. (11) The MR device 12 transmits the hand-held video data, captured data, etc. to the MR device 20.

[0026] (12) The MR device 20 receives hand-held video data, captured data, etc. from the MR device 12. (13) The MR device 20 displays the pointing gesture that represents the controller's intention as a hand model at the corresponding position. (14) The MR device 20 overlays the hand model onto the worker's actual subjective viewpoint. (15) The MR device 20 displays the image with the hand model overlaid on the display unit 25.

[0027] This communication between the supervisor's judgment and the worker's line of sight can correct misinterpretations of the supervisor's judgment by the worker. For example, if a worker mistakes a part located in a different spatial position than intended by the supervisor for the work object, the supervisor can notice this and correct it.

[0028] Figure 3 is a flowchart showing an example of the processing procedure in the controller's MR device 12. The MR device 12 transmits the controller's judgment information to the worker's device (MR device 20) (step S11). Next, the MR device 12 acquires the worker's eye information from the MR device 20 (step S12). Then, based on the eye information, the MR device 12 confirms that the controller's judgment and the worker's perception match (step S13).

[0029] Figure 4 is a flowchart showing an example of the processing procedure in the MR device 20 for the worker. The MR device 20 receives the controller's judgment information from the controller device (MR device 12) (step S21). Next, the MR device 20 transmits the worker's eye information to the MR device 12 (step S22). Then, based on the eye information, the MR device 20 confirms that the controller's judgment and the worker's perception match (step S23).

[0030] Figure 5 illustrates an example of a conversation between a worker and a supervisor. The worker on site shares a subjective viewpoint video with the supervisor and asks the supervisor via voice communication, "Which button should I press?" (Figure 5(a)). In response, the supervisor replies, for example, "It's here, above the green light." Simultaneously, a model of the supervisor's hand is overlaid on the supervisor's subjective viewpoint video (Figure 5(b)).

[0031] However, if there are multiple "green-lit areas," the supervisor and the worker may be looking at different locations, which can lead to miscommunication. In Figure 5, the areas indicated by coarse dot hatching are green displays, the downward-sloping hatching indicates yellow-green lights, and the upward-sloping hatching indicates cursors that require attention.

[0032] As shown in Figure 5(c), when the worker is looking at the display, there is no button above it, which raises the question, "There is no button above the green light..." and existing technology has no way to resolve this. However, in this embodiment, the screen shown in Figure 5(c) is also shared with the supervisor, so the supervisor can clearly understand the area the worker is paying attention to. Then, as shown in Figure 5(d), the supervisor can move the hand model to the right, point to the yellow-green button, and respond, "That's not it. It's a little further to the right, above the yellowish light." In this way, the discrepancy in mutual understanding can be resolved.

[0033] As described above, in this embodiment, eye information of the worker is transmitted to the controller as a hint for the controller to determine the worker's perception, thereby guaranteeing the accuracy of perception between the controller and the worker. In other words, in this embodiment, taking into account that humans obtain a lot of information from their vision, eye information is transmitted to each other in addition to the controller's hand model information. By overlaying both the hand model and the viewpoint onto the subjective viewpoint video, it becomes possible for the worker and the controller to mutually confirm whether the worker is correctly perceiving the controller's judgment.

[0034] In remote operation technology, where actions deemed necessary by the controller are communicated to the worker via an overlay display on the work object being viewed by the worker, there were cases where the worker could not correctly perceive the controller's judgment based solely on hand model information. In contrast, the first embodiment provides information that allows the controller to determine whether the worker has internally correctly perceived the communicated judgment. Therefore, it becomes possible for both the controller and the worker to determine whether the worker correctly perceived the controller's judgment after it has been communicated to them. Consequently, the worker's judgment can be accurately fed back to the controller, and the worker can correctly perceive the controller's judgment.

[0035] [Second Embodiment] <Example of transmitting the controller's cognitive state to the worker> Figure 6 is a functional block diagram showing an example of a work support system according to the second embodiment.

[0036] The MR device 12 includes a cognitive state estimation unit 34 instead of a worker gaze position cursor generation unit 32. The cognitive state estimation unit 34 estimates the controller's cognitive state from the controller's eye information.

[0037] The MR device 20 includes a controller cognitive state image generation unit 244 instead of the subjective viewpoint image gaze position calculation unit 241 shown in Figure 2. The controller cognitive state image generation unit 244 generates an image that visualizes the difficulty of the task as perceived by the controller. The image is, for example, modeled after the controller's face, and the difficulty increases as the controller grimaces.

[0038] (21) The subjective viewpoint camera 22b of the worker's MR device 20 captures the worker's subjective viewpoint video. (22) The MR device 20 transmits the subjective viewpoint video to the MR device 12. (23) The MR device 12 receives the subjective viewpoint video from the MR device 20 and displays it on the screen of the display unit 51.

[0039] (24) The hand-held camera 91 captures the operator pointing to equipment or other items that the operator wants the operator to operate, which are visible in the operator's subjective viewpoint video. (25) The MR device 12 captures the operator's hand-held video and converts it into a hand model. (27) The MR device 12 captures the operator's eyes using the eye-capture camera 92.

[0040] (28) The MR device 12 measures how the pupil diameter changes in response to the difficulty of the task predicted by the controller. (29) The MR device 12 transmits the degree of task difficulty perceived by the controller, based on the hand model and pupil diameter, to the MR device 20. (30) The MR device 20 receives the hand model generated by the MR device 12 and the degree of task difficulty perceived by the controller. (31) The MR device 20 represents the controller's intention (pointing gesture, hand model) as a 3D model.

[0041] (32) The MR device 20 visualizes the difficulty of the task as perceived by the supervisor by mimicking the supervisor's face, and by frowning as the difficulty increases. (33) The MR device 20 displays the hand model and cognitive state information on the display unit 25 in an overlay format on the worker's actual subjective viewpoint.

[0042] By exchanging the controller's judgment and cognitive state using MR devices 12 and 20, if the difficulty level anticipated by the controller differs from the actions communicated by the controller, the worker can notice this discrepancy and make corrections.

[0043] [Third Embodiment] <Example of Matching the Attention Coordinates of the Controller and the Worker> Figure 7 is a functional block diagram showing an example of a work support system according to the third embodiment. The MR device 12 includes a binocular camera 93, a subjective viewpoint video attention coordinate calculation unit 94, a worker / controller attention coordinate matching unit 95, and a warning unit 96, instead of the configuration in Figure 2 or Figure 6. The binocular camera 93 captures the left and right eyeballs of the controller and acquires eyeball direction data.

[0044] Instead of the configuration shown in FIG. 2 or FIG. 6, the MR device 20 includes a binocular imaging camera 22c, an attention coordinate calculation unit on subjective viewpoint image 245, an operator-controller attention coordinate matching unit 246, and a warning unit 247. The binocular imaging camera 22c images the eyeballs of the left and right eyes of the operator to acquire eyeball direction data. The attention coordinate calculation unit on subjective viewpoint image 245 calculates the line-of-sight coordinates of the operator in the subjective viewpoint based on the subjective viewpoint image and the eyeball direction data.

[0045] The operator-controller attention coordinate matching unit 246 exchanges data indicating the judgment of the controller with the MR device 12 via the network 100, and checks whether the attention coordinates of the operator and the controller are different between the MR device 12 and the MR device 20 at the time when the controller's judgment is transmitted to the operator. The warning unit 247 issues a warning when the attention coordinates are different from each other. In the above configuration, for example, the following procedure is executed.

[0046] (41) The binocular imaging camera 22c images the eyeballs of the left and right eyes of the operator to acquire eyeball direction data. (42) The subjective viewpoint camera 22b images the subjective viewpoint image of the operator to acquire subjective viewpoint image data. (43) The MR device 20 calculates the line-of-sight coordinates of the operator in the subjective viewpoint based on the subjective viewpoint image and the eyeball direction data. At this time, the MR device 20 calculates depth information where the two lines of sight coincide based on the convergence angle of the left and right eyes, and calculates attention coordinates with depth for the operator based on this depth information.

[0047] (44) The MR device 20 transmits the subjective viewpoint image and the attention coordinate information to the MR device 12. (45) The MR device 12 receives the subjective viewpoint image and the attention coordinate information from the MR device 20. (46) The MR device 12 displays only the subjective viewpoint image on the screen of the display unit 51 in a form with a sense of depth.

[0048] (47) The MR device 12 images the eyeballs of the controller who is watching the image on the display unit 51. (48) The MR device 12 calculates the line-of-sight coordinates of the controller from the subjective viewpoint based on the image on the screen and the eyeball direction. In this process, depth information at which the two lines of sight converge is calculated based on the vergence angles of the left and right eyes, thereby obtaining attention coordinates with depth information of the controller. (49) The MR device 12 transmits the attention coordinate information to the MR device 20.

[0049] (50) The MR device 12 uses the hand-held imaging camera 91 to capture images of the controller pointing at a device to be operated on the subjective viewpoint image of the worker. (51) The MR device 12 performs hand capture, and transmits information representing the controller's intention to the MR device 20. (52) The MR device 20 receives the hand capture information.

[0050] (53) The MR device 20 generates a 3D hand model reflecting the controller's intention. (54) The MR device 20 displays the 3D hand model in an overlaid manner on the worker's real-world subjective viewpoint. (55) Through the communication of the controller's judgment in these processes, the MR device 20 uses the worker-controller attention coordinate matching unit 246 to check whether the attention coordinates of the worker and the controller are different between the two devices at the time when the controller's judgment is transmitted to the worker. (56) The MR device 20 issues a warning if the attention coordinates are different. In addition, the warning unit 96 of the MR device 12 also issues a warning in the same manner. In this way, both the worker-side device and the controller-side device issue a warning (such as a warning sound), which can correct the worker's misperception of judgment.

[0051] Figure 8 is a block diagram showing an example of the hardware configuration of MR devices 12 and 20. As shown in Figure 8, MR devices 12 and 20 include a CPU (Central Processing Unit) 30A, a bridge circuit 102, memory 60, a GPU (Graphics Processing Unit) 30B connected to a display unit 51, a communication unit 70, storage 85, a USB connector 90, an input unit 40, and an output unit 50. The following description will use MR device 12 as an example, but the same applies to MR device 20.

[0052] The storage 85 is a non-volatile storage medium (block device), such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage 85 stores basic programs such as the OS (Operating System) 62 and device drivers, as well as a program 61 that causes the computer to function as an MR device 12.

[0053] Memory 60 includes ROM (Read Only Memory) and RAM (Random Access Memory). CPU 30A and GPU 30B are computing elements related to processor 30. CPU 30A mainly controls the MR device 12. GPU 30B mainly performs high-speed calculations related to image processing (such as multiply-accumulate operations). CPU 30A loads program 61 from storage 85 into memory 60 and executes it. GPU 30B does the same.

[0054] The bridge circuit 102 relays data exchange between the CPU 30A and GPU 30B and each component. The bridge circuit 102 intervenes between the input unit 40 and output unit 50 connected to the PCI (Peripheral Component Interconnect) bus, or a hardware device connected to the PCIe bus (not shown), and the CPU 30A and GPU 30B, relaying their communication with each other.

[0055] The communication unit 70 communicates with the MR device 20 via the network 100. The USB connector 90 connects USB devices, etc. For example, the program 61 may be installed on the MR device 12 via a USB device. The communication unit 70 connects the MR device 12 to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0056] The program 61 and various data are not limited to the storage 85; for example, they may be stored on a removable storage medium and read by the CPU 30A from a disk drive or the like. Alternatively, they may be stored on another computer connected via a communication network and read by the CPU 30A via the communication unit 70.

[0057] It should be noted that this invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0058] 11...Camera 12...MR device 20...MR device 22...Camera 22a...Eyeball camera 22b...Subjective viewpoint camera 22c...Binocular camera 23...Communication unit 25...Display unit 30...Processor 30A...CPU 31...Hand capture unit 32...Worker gaze position cursor generation unit 33...Image superposition unit 34...Cognitive state estimation unit 40...Input unit 50...Output unit 51...Display unit 60...Memory 61...Program 70...Communication unit 85...Storage 90...USB connector 91...Hand-camera 92...Eyeball camera 93...Binocular camera 94...Attention coordinate calculation unit on subjective viewpoint video 95...Worker / controller attention coordinate matching unit 96...Warning unit 100...Network 102...Bridge circuit 241...Gaze position calculation unit on subjective viewpoint video 242...Hand model generation unit 243...Video overlay unit 244...Controller cognitive state image diagram generation unit 245...Subjective viewpoint video attention coordinate calculation unit 246...Worker / controller attention coordinate matching unit 247...Warning unit.

Claims

1. A work support system comprising: a first MR (Mixed Reality) device; a second MR device capable of communicating with the first MR device via a network; and a subjective viewpoint camera that captures subjective viewpoint video of a first user wearing the first MR device and acquires subjective viewpoint video data, wherein the first MR device comprises: an eyeball imaging camera that captures the eyeballs of the first user and acquires gaze coordinate data; and a transmission unit that transmits the subjective viewpoint video data of the first user and the gaze coordinate data to the second MR device via the network; and the second MR device comprises: a receiving unit that receives the subjective viewpoint video data of the first user and the gaze coordinate data; a cursor generation unit that generates cursor information indicating the first user's attention points from the received gaze coordinate data; and a display unit that superimposes the cursor information onto the received subjective viewpoint video data of the first user and displays it in the field of view of a second user wearing the second MR device.

2. The work support system according to claim 1, wherein the first MR device further comprises an attention coordinate calculation unit that calculates depth information in which the lines of sight of both eyes coincide from the convergence angles of both eyes of the first user obtained from the line-of-sight coordinate data, and calculates the attention coordinates of the first user with respect to depth.

3. The work support system according to claim 1, further comprising a hand-camera for capturing images of the hands of a second user and acquiring hand-camera video data, wherein the second MR device includes a transmission unit for transmitting the hand-camera video data to the first MR device, and the first MR device includes a reception unit for receiving the hand-camera video data, a hand model generation unit for generating a hand model indicating the object to be instructed by the second user from the received hand-camera video data, and a display unit for superimposing the hand model onto the subjective viewpoint video data of the first user and displaying it in the field of view of the first user.

4. A work support method for a work support system comprising a first MR (Mixed Reality) device and a second MR device capable of communicating with the first MR device via a network, the work support method comprising: the first MR device transmitting subjective viewpoint video data acquired by a subjective viewpoint camera that captures subjective viewpoint video of a first user wearing the first MR device, and gaze coordinate data acquired by an eyeball camera that captures the eyeballs of the first user, to the second MR device via the network; the second MR device receiving the subjective viewpoint video data and the gaze coordinate data of the first user; the second MR device generating cursor information indicating the first user's attention points from the received gaze coordinate data; and the second MR device superimposing the cursor information onto the received subjective viewpoint video data of the first user and displaying it in the field of view of a second user wearing the second MR device.

Images