Work instruction display device and program

Abstract

This work instruction display device is provided with a processor that executes a process for acquiring a gesture of an instructor instructing a worker to perform work, a process for generating a model hologram (first hologram) corresponding to the acquired gesture of the instructor, a process for inputting, in accordance with an operation of the instructor, a processing intention for processing a portion of the model hologram corresponding to the type of the work, a process for generating, in accordance with the input processing intention, a processed hologram (second hologram) obtained by processing the model hologram on the basis of a processing definition in which information about the portion of the model hologram to be processed is combined with information about a processing method, and a process for outputting the processed hologram to a terminal of the worker.

Description

Work Instruction Display Device and Program

[0001] An embodiment of the present invention relates to a work instruction display device and a program.

[0002] In recent years, the number of workers who perform collaborative work remotely, such as remote work, has been increasing.

[0003] Here, the collaborative work refers to collaborative work (hereinafter, "remote collaborative work") in which a controller (hereinafter, "instructor") gives instructions to a worker by utilizing a call or local video from a remote location.

[0004] Examples of remote collaborative work include: Example 1: Fault repair work on servers and OA devices; Example 2: Fault repair work on outdoor machines.

[0005] In remote collaborative work, a method has been considered in which a worker wears a transmissive type HMD (Head Mount Display) such as smart glasses with a camera and shares video with an instructor remotely.

[0006] In remote collaborative work, a method has been devised in which hand model information acquired from an instructor is superimposed and displayed in the worker's field of view on smart glasses to efficiently convey the work content.

[0007] Shunsuke Ichihara, Yusuke Suzuki, "Field Evaluation Experiment of a Remote Work Support System Using Hand Gesture and Line Drawing Function," Information Processing Society of Japan Interaction 2017, 1-502-13, pp.154-157, Feb. 2017. Shohei Yamada, Naiwala P. Chandrasiri, "Verification of Work Efficiency in Remote Work Support Using Smart Glasses and Hand Gestures," Journal of the Institute of Image Electronics Engineers of Japan, vol.47, no.4, pp.401-404, April 2018.

[0008] However, when a worker without know-how receives instructions from an instructor from a remote location and performs remote collaborative work, due to reasons such as the fact that many of the equipment, tools, and devices to be worked on by the worker require careful handling, it is difficult to convey fine expressions such as the degree of force, and there is a problem that the work target indicated by the words of the instruction may not be conveyed to the worker.

[0009] This invention has been made in view of the above problems, and aims to provide a work instruction display device and program that enables an instructor to convey detailed expressions such as the amount of force to be applied when giving instructions to a worker from a remote location, and to correctly convey to the worker the work object that the instruction refers to.

[0010] The work instruction display device according to the present invention includes a processor that performs the following: a process of acquiring the gesture of an instructor who instructs an operator to perform a task; a process of generating a first hologram corresponding to the acquired gesture of the instructor; a process of inputting a processing intention to process a portion of the first hologram corresponding to the type of work, in response to the operation of the instructor; a process of generating a second hologram obtained by processing the first hologram based on a processing definition that combines information on the portion of the first hologram to be processed and information on the processing method, in response to the input processing intention; and a process of outputting the second hologram to the operator's terminal.

[0011] According to the work instruction display device of the present invention, when an instructor gives instructions to a worker from a remote location, it becomes possible to convey detailed expressions such as the amount of force to be applied, and to correctly convey to the worker the work object that the instruction refers to.

[0012] Figure 1 is a functional block diagram showing an example of the overall configuration of a work instruction display system 1 according to a first embodiment of the work instruction display device and program of the present invention. Figure 2 is a diagram showing an example of a processing definition defined in the processing definition unit 122 of the hologram generation unit 12 of the instructor terminal 10 (work instruction display device) of the work instruction display system 1. Figure 3 is a diagram showing an example of a processing definition including key bindings defined in the processing definition unit 122 of the hologram generation unit 12 of the instructor terminal 10 (work instruction display device) of the work instruction display system 1. Figure 4 is a flowchart showing an example of processing performed by the instructor terminal 10 (work instruction display device) and the worker terminal 20 of the work instruction display system 1. Figure 5 is a flowchart showing the hologram generation process (S14) included in the processing performed by the instructor terminal 10. Figure 6 is a diagram showing a processing hologram Hg1 of a hand model corresponding to the instructor's gesture, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and generated based on a processing definition that emphasizes the fingernails N (yellow). Figure 7 shows a hologram of a hand model corresponding to the gesture of an instructor, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and is a processed hologram Hg2 generated based on a processing definition that emphasizes (rotates a1) the fingertip F. Figure 8 shows a hologram of a hand model corresponding to the gesture of an instructor, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and is a processed hologram Hg3 generated based on a processing definition that emphasizes (turns into a work tool T and rotates a2) the fingertip F. Figure 9 is a functional block diagram showing an example of the overall configuration of a work instruction display system 1 according to a second embodiment of the work instruction display device and program of the present invention. Figure 10 is a block diagram showing an example of the hardware configuration of a work instruction display system 1 according to an embodiment of the present invention.

[0013] Hereinafter, embodiments of the work instruction display device and program of the present invention will be described with reference to the drawings.

[0014] (Configuration of the First Embodiment) Figure 1 is a functional block diagram showing an example of the overall configuration of a work instruction display system 1 according to the first embodiment of the work instruction display device and program of the present invention.

[0015] The work instruction display system 1 includes, for example, an instructioner terminal 10 (work instruction display device) of an instructioner DM that gives work instructions to a worker WM from a remote location, and a worker terminal 20 of a worker WM that performs work on the work target Tw in response to instructions from the instructioner DM.

[0016] Both the instructor terminal 10 and the worker terminal 20 are implemented by having the processor execute the functions of each part shown in Figure 1 according to a program (through the cooperation of hardware and software) in an information processing device such as an HMD (Head Mount Display) (wearable device), PC (Personal Computer), or tablet terminal, which is equipped with a processor (CPU: Central Processing Unit). Both the instructor terminal 10 and the worker terminal 20 may be implemented by combining multiple types of information processing devices, such as an HMD and a PC, or by implementing them with a single information processing device.

[0017] In the first embodiment, the instructor terminal 10 is composed of an HMD1 and a PC, and the worker terminal 20 is composed of an HMD2.

[0018] The instruction terminal 10 includes a gesture acquisition unit 11, a hologram generation unit 12, a synthesis unit 13, a display unit 14, and a processing intent input unit 15.

[0019] The hologram generation unit 12 includes a model hologram generation unit 121, a processing definition unit 122, a hologram processing unit 123, and an output unit 124.

[0020] The worker terminal 20 has a camera 21, a synthesis unit 22, and a display unit 23.

[0021] In the instructor terminal 10, the gesture acquisition unit 11 acquires the three-dimensional structure (spatial structure) of the instructor DM's hand using a camera or sensor, etc., as gesture (hand model) information Ge. The gesture information Ge is represented by a set of coordinates, etc.

[0022] The hologram generation unit 12 generates and outputs hologram information Hg corresponding to the gesture information Ge acquired by the gesture acquisition unit 11. The hologram information Hg is represented by a set of coordinates, etc.

[0023] In the hologram generation unit 12, the model hologram generation unit 121 generates a model hologram (first hologram) corresponding to the gesture information Ge. The model hologram is represented by a set of coordinates, etc.

[0024] The hologram processing unit 123 processes the model hologram using the processing definition defined in the processing definition unit 122 (see Figure 2 or 3) to generate a processed hologram (second hologram) corresponding to the processing intent ip, when the processing intent input unit 15 receives a processing intent ip specified by the processing intent input unit 123 in accordance with the specified intent id of the PC operation (key operation, etc.) of the instructioner DM. The processed hologram is represented by a set of coordinates, etc.

[0025] If no processing intent ip is input via the processing intent input unit 15, the output unit 124 outputs the model hologram generated by the model hologram generation unit 121 as hologram information Hg to the synthesis unit 13 of the instruction terminal 10 and the synthesis unit 22 of the worker terminal 20. If a processing intent ip is input, the output unit 124 outputs the processing hologram generated by the hologram processing unit 123 as hologram information Hg to the synthesis unit 13 of the instruction terminal 10 and the synthesis unit 22 of the worker terminal 20.

[0026] The hologram information Hg, as a processed hologram, is represented by a set of pairs of coordinates and the color of the coordinates of the emphasized part, for example, in the case of processed hologram Hg1 (see Figure 6) generated based on a processing definition that emphasizes (yellow) the fingernail N.

[0027] Furthermore, for example, in the case of a processed hologram Hg2 (see Figure 7) generated based on a processing definition that emphasizes (rotates a1) the fingertip F, it is represented by a set of pairs of coordinates and the movement (change) of the coordinates of the emphasized part. In this case, an arrow a1 (drawing line) corresponding to the movement (change) of the coordinates of the emphasized part may be added.

[0028] Furthermore, for example, in the case of a processed hologram Hg3 (see Figure 8) generated based on a processing definition that emphasizes the fingertip F (by converting it to a work tool T and rotating it a2), it is represented by a set of coordinates in which the emphasized part is replaced (modified) with the work tool T. In this case, an arrow a2 (drawing line) corresponding to the movement specific to the work tool T of the emphasized part may be added.

[0029] The addition of arrow an (drawing line) may be included in the processing intent ip in the processing intent input unit 15 and the processing definition in the processing definition unit 122, corresponding to the specified id of the intention to process the model hologram based on the PC operation (key operation, etc.) by the instructioner DM.

[0030] The synthesis unit 13 synthesizes (overlays the hologram on the image) the video Pw of the work environment, including the work target Tw, which is input (received) from the worker terminal 20, and the hologram (Hg) output from the hologram generation unit 12.

[0031] The display unit 14 displays a video Pw of the work environment on the HMD1, on which a hologram (Hg) is superimposed. The video Pw of the work environment includes, for example, the work object Tw such as a server machine and the location such as the desk on which the work object Tw is placed, but the hands of the worker WM may or may not be visible.

[0032] In the worker terminal 20, the camera 21 acquires video (image) Pw of the work environment, including the worker WM's work target Tw.

[0033] The synthesis unit 22 synthesizes (overlays the hologram on the image) the video Pw of the work environment including the work target Tw acquired by the camera 21 and the hologram (Hg) generated by the hologram generation unit 12 of the instruction terminal 10.

[0034] The display unit 23 displays an image Pw of the work environment, on the HMD2, with a hologram (Hg) superimposed on it. When the worker WM places their hand on the work environment, their hand is included in the image Pw of the work environment.

[0035] Furthermore, the coordinate systems of the display unit 23 of the worker terminal 20, the display unit 14 of the instructor terminal 10, and the gesture acquisition unit 11 of the instructor terminal 10 are the same. The video Pw of the work environment, on which the hologram (Hg) corresponding to the gesture information Ge of the instructor DM is superimposed, is the same on both the display unit 14 of the instructor terminal 10 and the display unit 23 of the worker terminal 20.

[0036] Figure 2 shows an example of a processing definition defined in the processing definition unit 122 of the hologram generation unit 12 of the instructor terminal 10 (work instruction display device) of the work instruction display system 1.

[0037] The processing definition unit 122 stores a combination of information about the part of the hologram (Hg) to be emphasized (processed) (processed area) and information about the processing method (processing treatment) for that processed area as a single processing definition. The number of processing definitions corresponding to the type of processing intent ip entered from the processing intent input unit 15 according to the processing intent specification id by the instructioner DM is stored.

[0038] As an example of a processing definition, the processing area of ​​the hologram may be stored as [whole], [fingertip (part of the specified)], [finger (part of the specified)], [nail (part of the specified)], etc., and the processing steps may be stored as [enlarge (expand)], [reduce (shrink)], [toolify (modify)], [color processing (exaggerate)], [arrow (add)], etc. The direction of the [arrow (add)] and whether the [arrow] is added statically or dynamically may also be selectively specified in the processing intent specification ID by the instruction DM.

[0039] Furthermore, the method for specifying the processing intent ID based on the PC operation (key operations, etc.) by the instruction DM can be, for example, by selectively selecting from a list of processing definitions displayed on the PC, by keybinding to a key on the PC for selective selection, or by any other method.

[0040] Figure 3 shows an example of a machining definition including key bindings defined in the machining definition unit 122 of the hologram generation unit 12 of the instruction terminal 10 (work instruction display device) of the work instruction display system 1.

[0041] For example, as shown in FIG. 3, by associating each machining definition defined in the machining definition unit 122 with the numeric keys (ten-key) of the PC and performing key binding, the instructor DM can selectively operate the numeric keys of the PC to easily specify the designation ID of the desired machining for machining the hologram (model hologram) Hg.

[0042] In the work instruction display system 1 of the first embodiment configured as described above, the worker WM moves his own hand shown in the video (Pw) so as not to deviate in accordance with the hologram (Hg) corresponding to the gesture (hand model) (Ge) of the instructor DM projected on the display unit 23 (HMD2) of the worker terminal 20, thereby allowing the instructor DM to guide the movement of the worker WM.

[0043] At this time, the worker WM can easily notice the content of the display emphasized (machined) according to the machining intention ip of the instructor DM in the hologram (Hg), and can proceed with the work in accordance with the content of the display. The instructor DM can guide the worker WM while correctly conveying the detailed work content and work target intended by himself to the worker WM.

[0044] (Operation of the First Embodiment) Next, the operation of the work instruction display system 1 of the first embodiment will be described.

[0045] FIG. 4 is a flowchart showing an example of the processes executed by the instructor terminal 10 (work instruction display device) and the worker terminal 20 of the work instruction display system 1.

[0046] FIG. 5 is a flowchart showing the hologram generation process (S14) included in the process executed by the instructor terminal 10.

[0047] Here, it is assumed that the worker WM is an unskilled person with little work experience on the work target Tw, and the instructor DM is a skilled person with a lot of work experience on the work target Tw.

[0048] In the worker terminal 20, first, the camera 21 acquires a video Pw of the work environment including the work target Tw of the worker WM (step S21). The video Pw of the work environment acquired by the camera 21 is output (transmitted) to the instructor terminal 10 (step S22).

[0049] The worker terminal 20 determines whether or not hologram information Hg has been input (received) from the instructor terminal 10 (step S23). If hologram information Hg is not input (received) from the instructor terminal 10 (step S23 (No)), the synthesis unit 22 outputs the video Pw of the work environment acquired by the camera 21 to the display unit 23 as is, and the display unit 23 displays the video Pw of the work environment output from the synthesis unit 22 on the HMD2 (step S24).

[0050] The worker WM performs tasks on the target Tw while viewing the video Pw of the work environment displayed on the HMD2.

[0051] In the instructor terminal 10 (work instruction display device), first, when the video Pw of the worker WM's work environment, which is output (transmitted) from the worker terminal 20, is input (received), the synthesis unit 13 outputs the input video Pw of the work environment as is to the display unit 14, and the display unit 14 displays the video Pw of the work environment output from the synthesis unit 13 on the HMD1 (step S11).

[0052] The gesture acquisition unit 11 starts acquiring gesture (hand model) information Ge corresponding to the hand movements of the instructor DM (step S12).

[0053] The instructor DM, while viewing the video Pw of the work environment, including the worker's hands working on the target Tw, displayed on the HMD1, moves their own hands within the gesture acquisition range of the gesture acquisition unit 11's camera and sensors in order to guide the worker WM's work (hand movements).

[0054] When the gesture acquisition unit 11 acquires gesture information Ge corresponding to the hand movements of the instructor DM (step S13 (Yes)), the hologram generation unit 12 executes the process of generating hologram information Hg corresponding to the gesture information Ge (see Figure 5) (step S14).

[0055] In the hologram generation unit 12, the model hologram generation unit 121 generates a model hologram (first hologram) corresponding to the gesture information Ge acquired by the gesture acquisition unit 11 (step S141).

[0056] Here, the hologram generation unit 12 determines whether or not a processing intention ip corresponding to the specified id of the intention to process the model hologram based on the PC key operations etc. by the instructioner DM has been input from the processing intention input unit 15 (step S142).

[0057] If no processing intent ip is input from the processing intent input unit 15 in the hologram generation unit 12 (step S142 (No)), the output unit 124 outputs the model hologram generated by the model hologram generation unit 121 as hologram information Hg to the synthesis unit 13 in the instruction terminal 10, and also outputs (transmits) it to the operator terminal 20 (steps S143, S15).

[0058] The synthesis unit 13 of the instructor terminal 10 superimposes (step S16) a hologram (Hg) corresponding to the gesture (Ge) of the instructor DM, which is input (received) from the hologram generation unit 12, onto the video Pw of the work environment, which is input (received) from the worker terminal 20 and displayed on the HMD1.

[0059] The display unit 14 displays the video Pw of the work environment, which has a hologram (Hg) superimposed on it, on the HMD1 (step S17).

[0060] The instructor (DM) guides the worker's (WM) hand movements while superimposing a hologram (Hg) corresponding to the worker's (WM's) hand movements onto the video (Pw) of the work environment, including the worker's hands, displayed on the HMD1.

[0061] When the worker terminal 20 receives a hologram (Hg) (in this case, a model hologram) corresponding to the gesture (Ge) of the instructor DM output (transmitted) from the hologram generation unit 12 of the instructor terminal 10 (step S23 (Yes)), the synthesis unit 22 superimposes the hologram (Hg) (in this case, a model hologram) received from the instructor terminal 10 onto the video Pw of the work environment acquired from the camera 21 and displayed on the HMD2, and synthesizes them (step S25).

[0062] The display unit 23 displays the video Pw of the work environment, which is a composite of a hologram (Hg) (in this case, a model hologram), on the HMD2 (step S26).

[0063] The worker (WM) performs the task by moving their own hands, which are included in the video (Pw) displayed on the HMD2, while being guided by a hologram (Hg) (in this case, a model hologram) corresponding to the gestures (Ge) of the instructor (DM) superimposed on the video (Pw) of the work environment.

[0064] Here, if the instructor DM feels that it is difficult to correctly convey the detailed work content and target of the work intended by the instructor DM to the worker WM for the work in which the instructor DM is currently guiding the worker WM, for example, by simply compositing a hologram (Hg) (model hologram) corresponding to the instructor's gesture (Ge) onto the video Pw of the work environment and displaying it on the worker terminal 20 (HMD2), the instructor DM operates the PC to specify an intention id to process the model hologram in order to specify a desired processing definition (processing definition according to the type of work) defined in the processing definition unit 122 (see Figure 2 or Figure 3).

[0065] In the hologram generation unit 12 of the instruction terminal 10, when a processing intention ip corresponding to the specified id for the intention to process the model hologram is input from the processing intention input unit 15 (step S142 (Yes) in Figure 5), the hologram processing unit 123 processes the model hologram using the processing definition defined in the processing definition unit 122 corresponding to the input processing intention ip, and generates a processed hologram (second hologram) corresponding to the processing intention ip, for example, as shown in Figures 6 to 8. The hologram processing unit 123 outputs the generated processed hologram as hologram information Hg to the synthesis unit 13 in the instruction terminal 10 and also outputs (transmits) it to the operator terminal 20 (steps S144, S15).

[0066] Figure 6 shows a hologram of a hand model corresponding to the gesture of the instructor, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and is a processed hologram Hg1 generated based on a processing definition that emphasizes the fingernails N (yellow).

[0067] Figure 7 shows a hologram of a hand model corresponding to the gesture of the instructor, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and is a processed hologram Hg2 generated based on a processed definition that emphasizes (rotates a1) the fingertip F.

[0068] Figure 8 shows a hologram of a hand model corresponding to the gesture of the instructor, generated according to the hologram generation process (S14) performed by the instructor terminal 10, and is a processed hologram Hg3 generated based on a processing definition that emphasizes the fingertip F (converted to a work tool T and rotated a2).

[0069] The synthesis unit 13 of the instructor terminal 10 superimposes a processed hologram (for example, Hg1, Hg2, or Hg3) generated by the hologram processing unit 123 of the hologram generation unit 12 onto the video Pw of the work environment, which is input (received) from the worker terminal 20 and displayed on the HMD1, and synthesizes them (step S16).

[0070] The display unit 14 displays the image Pw of the work environment, into which the processed hologram (Hgn) has been synthesized, on the HMD1 (step S17).

[0071] The instructor (DM) guides the hand movements of the worker (WM) while displaying a processed hologram (Hgn) corresponding to the movements of the worker's (WM) hands, which is synthesized and enhanced (processed) by expanding or modifying (processing) the entire or any part of the work environment (Pw) displayed on the HMD1.

[0072] When the worker terminal 20 receives a processing hologram (Hgn) corresponding to the gesture (Ge) of the instructor DM output (transmitted) from the hologram generation unit 12 of the instructor terminal 10 (step S23 (Yes)), the synthesis unit 22 superimposes the processing hologram (Hgn) received from the instructor terminal 10 onto the video Pw of the work environment acquired from the camera 21 and displayed on the HMD2, and synthesizes them (step S25).

[0073] The display unit 23 displays the image Pw of the work environment, on which the processed hologram (Hgn) has been synthesized, on the HMD2 (step S26).

[0074] The worker (WM) performs the task by moving their own hands, which are included in the video (Pw) displayed on the HMD2, while being guided by a hologram (Hg) (in this case, a processing hologram Hgn) corresponding to the gestures (Ge) of the instructor (DM) superimposed on the video (Pw) of the work environment.

[0075] In this case, the hologram (Hg) of the hand model corresponding to the gestures of the instructor DM (in this case, the processing hologram Hgn) displayed on the worker WM's HMD2, which is superimposed on the video Pw of the work environment, is expanded or modified and emphasized (processed) in whole or in any part, based on the processing definition of the processing intent (ip) specified (id) by the instructor DM according to the content (type of work) that the worker WM is working on the work object Tw. This allows the instructor DM to correctly convey to the worker WM the detailed work content and work object, such as the amount of force to be applied, while guiding the worker WM's work.

[0076] In other words, if the task performed by worker WM on the work object Tw is, for example, removing or hooking the tabs on the cover of a server or office equipment, then, as shown in Figure 6, a processed hologram Hg1 is generated that emphasizes (processes) the color of the index finger and nail N of the hand model (model hologram) corresponding to the gesture of the instructor DM. This processed hologram Hg1 is then superimposed (overlaid) onto the video of the work environment Pw and displayed on the worker WM's HMD2. This allows for accurate communication of the instructor DM's intentions to the worker WM, which would be difficult to convey with just a model hologram corresponding to the instructor DM's gestures.

[0077] Furthermore, if the task performed by worker WM on the work object Tw is, for example, turning screws on the cover of a server or office equipment, as shown in Figure 7, a processed hologram Hg2 is generated in which the index finger F of the hand model (model hologram) corresponding to the gesture of the instructor DM is emphasized (processed) to rotate a1. This processed hologram Hg2 is then superimposed (overlaid) onto the video of the work environment Pw and displayed on the worker WM's HMD2. This allows the instructor DM to easily convey their intentions to the worker WM without using actual tools such as screwdrivers.

[0078] Furthermore, if the work performed by the worker WM on the work object Tw involves, for example, the use of a tool, as shown in Figure 8, a processed hologram Hg3 is generated by changing (replacing) the index finger F of the hand model (model hologram) corresponding to the gesture of the instructor DM with a rotating work tool T (in this case, a driver), and emphasizing (processing) it. This processed hologram Hg3 is then composited (superimposed) onto the video of the work environment Pw and displayed on the worker WM's HMD2. In this way, the instructor DM can communicate their intentions to the worker WM more accurately and easily without using actual tools.

[0079] Furthermore, the types of processing definitions defined (stored) in the processing definition unit 122 of the hologram generation unit 12, and the types of processed holograms Hgn that can be generated from the model hologram based on said processing definitions, are not limited to the types described above. Any processing that enhances the entire hand model (model hologram) or any part (processing area) corresponding to the gesture of the instructioner DM into any shape is acceptable.

[0080] (Second Embodiment) Figure 9 is a functional block diagram showing an example of the overall configuration of a work instruction display system 1 according to the second embodiment of the work instruction display device and program of the present invention.

[0081] The work instruction display system 1 of the second embodiment is a simplified version of the work instruction display system 1 of the first embodiment shown in Figure 1. The instructor terminal 101 (PC) (work instruction display device) has the same configuration as the instructor terminal 10 of the first embodiment, but without the HMD1 and the composite unit 13. The worker terminal 201 (PC) has the same configuration as the worker terminal 20 of the first embodiment, but without the HMD2, camera 21 and the composite unit 22.

[0082] In the work instruction display system 1 of the second embodiment, the work environment status, including the work target Tw, for the worker WM is grasped by the instructioner DM through telephone, text communication, etc.

[0083] The instructor DM, while understanding the work environment (work target Tw), including the work status of the worker WM, through telephone or text communication, checks the display unit 14 on the instructor terminal 101 (PC) (work instruction display device) to display a hologram (model hologram or processed hologram) Hg of the hand model that is generated in response to the instructor's gestures.

[0084] The worker WM performs work on the target Tw while confirming the hand model hologram (model hologram or processed hologram) Hg, which corresponds to the gesture of the instructor DM, generated at the instructor terminal 101 and input (received) at the worker terminal 201 (PC), displayed on the display unit 23.

[0085] On the instructor terminal 101 (PC), if the instructor DM feels that it is difficult to correctly convey the detailed work content or target of the work to the worker WM by simply displaying a hologram (Hg) (model hologram) corresponding to their own gesture (Ge) on the display unit 23 of the worker terminal 201 (PC), as in the first embodiment, they can specify a desired processing definition defined in the processing definition unit 122 (see Figure 2 or Figure 3) by operating the PC and specifying an ID indicating their intention to process the model hologram.

[0086] Then, the hologram processing unit 123 of the hologram generation unit 12 processes the model hologram using the processing definition defined in the processing definition unit 122 in response to the processing intent ip input from the processing intent input unit 15, similar to the first embodiment, and generates a processed hologram corresponding to the processing intent ip, for example, as shown in Figures 6 to 8. The hologram processing unit 123 then outputs the generated processed hologram as hologram information Hg to the display unit 14 in the instruction terminal 101 (PC) for display, and also outputs (transmits) it to the worker terminal 201 (PC) for display on the worker WM's display unit 23.

[0087] The worker WM performs the task by moving their hands, guided by a hologram (Hg) (in this case, a processing hologram Hgn) that corresponds to the gesture (Ge) of the instructor DM, which is displayed on the display unit 23.

[0088] In this case, the hand model hologram (Hg) (here, the processing hologram Hgn) displayed on the display unit 23 of the worker WM, which corresponds to the gesture of the instructor DM, is expanded or modified and emphasized (processed) in whole or in any part, based on the processing definition of the processing intent (ip) specified (id) by the instructor DM according to the content (type of work) that the worker WM will perform on the work object Tw. Therefore, similar to the first embodiment, the instructor DM can guide the worker WM's work while correctly conveying to the worker WM the detailed work content and work object, such as the amount of force intended by the instructor DM.

[0089] Figure 10 is a block diagram showing an example of the hardware configuration of a work instruction display system 1 according to an embodiment of the present invention.

[0090] In the example shown in Figure 10, the instruction terminal 10 (work instruction display device) according to the above embodiment is composed of, for example, a server computer, a personal computer, an HMD (head-mounted display), or a combination thereof, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0091] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network (network) NW. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0092] The input / output interface 113 is connected to an input device 200 and an output device 300 used by a user (instructor DM) or the like, which are attached to the instruction terminal 10.

[0093] The input / output interface 113 receives operation data entered by a user or the like through an input device 200 such as a keyboard, touch panel, touchpad, or mouse, and outputs the output data to an output device 300, including a display device using liquid crystal or organic EL (electroluminescence), for display. The input device 200 and output device 300 may be devices built into the instructor terminal 10 (work instruction display device), or they may be input and output devices of other information processing devices (information processing terminals) that can communicate with the instructor terminal 10 via a network NW.

[0094] The program memory 111B is a non-temporary tangible storage medium in which a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), is used in combination with another non-volatile memory such as ROM (Read Only Memory), and stores the programs necessary to execute various control processes according to the embodiment.

[0095] The data memory 112 is a tangible storage medium that, for example, uses a combination of the aforementioned non-volatile memory and volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and generated during the process of various operations.

[0096] An instruction terminal 10 (work instruction display device) according to an embodiment of the present invention may be configured as a data processing device having at least a gesture acquisition unit 11, a hologram generation unit 12, a synthesis unit 13, a display unit 14, and a processing intent input unit 15 as software processing function units, as shown in Figure 1.

[0097] Each information storage unit (which may include the processing definition unit 122) used as a working memory by each part of the instruction terminal 10 can be configured using the data memory 112 shown in Figure 10. However, these storage areas are not essential to the instruction terminal 10, and may be, for example, areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or in a storage device such as a database server located in the cloud.

[0098] The processing functions in the gesture acquisition unit 11, hologram generation unit 12, synthesis unit 13, display unit 14, and processing intent input unit 15 can all be implemented by having a hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0099] Furthermore, the worker terminal 20 according to the above embodiment is composed of, for example, a personal computer (PC), an HMD, or a combination thereof, and, like the instructor terminal 10, has a hardware processor (111A) such as a CPU. And, similar to the instructor terminal 10, a program memory (111B), a data memory (112), an input / output interface (113), and a communication interface (114) are connected to this hardware processor (111A) via a bus (115).

[0100] The worker terminal 20 can be configured as a data processing device having at least the camera 21, synthesis unit 22, and display unit 23 shown in Figure 1, as a software-based processing unit.

[0101] The processing functions in the camera 21, the synthesis unit 22, and the display unit 23 can all be realized, similar to the instruction terminal 10, by having a hardware processor (111A) read and execute a program stored in the program memory (111B).

[0102] (Summary of Embodiments) According to the work instruction display system 1 of the embodiment, the worker terminal 20 acquires video Pw of the work environment including the work target Tw of the worker WM using the camera 21, displays it on the worker WM's HMD2 via the synthesis unit 22, and outputs (transmits) it to the instructor terminal 10 (work instruction display device). The instructor terminal 10 displays the video Pw of the work environment input (received) from the worker terminal 20 on the instructor DM's HMD1 via the synthesis unit 13. When the instructor DM makes a hand gesture to guide the worker WM's work based on the video Pw of the work environment, the hologram generation unit 12 generates a model hologram (Hg) (first hologram) of the instructor DM's hand model according to the gesture information Ge acquired by the gesture acquisition unit 11. The model hologram (Hg) of the instructor DM is superimposed (overlaid) onto the video Pw of the work environment by the synthesis unit 13 of the instructor terminal 10 and displayed on the HMD1. It is also output (transmitted) to the worker terminal 20, where the synthesis unit 22 of the worker terminal 20 superimposes (overlays) onto the video Pw of the work environment and displays on the HMD2. The worker WM moves their hands in accordance with the model hologram (Hg) of the instructor DM and performs the work.

[0103] In the instructor terminal 10, if the instructor DM specifies an intention to process the model hologram (Hg) according to the type of work, for example, in order to correctly convey the content of the work to the worker WM, the hologram generation unit 12 processes the model hologram (Hg) using the processing definition defined in the processing definition unit 122 in response to the processing intention ip input by the processing intention input unit 15, and generates a processed hologram (Hgn) (second hologram) according to the processing intention ip. The processed hologram (Hgn) is composited (superimposed) on the video Pw of the work environment by the synthesis unit 13 of the instructor terminal 10 and displayed on the HMD1, and is also output (transmitted) to the worker terminal 20, where it is composited (superimposed) on the video Pw of the work environment by the synthesis unit 22 of the worker terminal 20 and displayed on the HMD2.

[0104] In this case, the processing hologram (Hgn) is expanded or modified and emphasized (processed) in whole or in any part thereof, based on the processing definition of the processing intent (ip) specified (id) by the supervisor DM according to the content (type of work) that the worker WM will perform on the work object Tw. Therefore, the supervisor DM can guide the worker WM's work while correctly conveying to the worker WM the detailed work content and work object, such as the amount of force they intend to apply.

[0105] Therefore, according to the work instruction display system 1 of this embodiment, when an instructor gives instructions to a worker from a remote location, it is possible to suppress misunderstandings such as errors and mistakes, to convey detailed expressions such as the amount of force to be applied, and to correctly convey to the worker the work object that the instruction refers to.

[0106] The methods described in each of the embodiments above can be stored as programs (software means) that can be executed by a computer, for example, on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memory, etc.), and can also be transmitted and distributed via communication media.

[0107] Furthermore, the program stored on the medium includes a configuration program that configures the computer with software means (including not only the executable program but also tables and data structures) to be executed by the computer. The computer that implements this device reads the program recorded on the recording medium and, if necessary, constructs the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. In this specification, the recording medium is not limited to distribution media, but also includes storage media such as magnetic disks and semiconductor memory installed inside the computer or in devices connected via a network.

[0108] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the embodiments described above include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed.

[0109] For example, if the problem can be solved and the desired effect can be obtained even if some of the constituent elements shown in the embodiment are removed, then the configuration with these removed constituent elements can be extracted as an invention.

[0110] 1...Work instruction display system DM...Instructor 10...Instructor terminal (work instruction display device) 11...Gesture acquisition unit 12...Hologram generation unit 121...Model hologram generation unit 122...Processing definition unit 123...Hologram processing unit 124...Output unit 13...Composition unit 14...Display unit 15...Processing intent input unit WM...Worker 20...Worker terminal 21...Camera 22...Composition unit 23...Display unit HMD...Head-mounted display PC...Personal computer

Claims

1. A work instruction display device comprising a processor that performs the following steps: acquiring gestures of an instructor who instructs a worker to perform a task; generating a first hologram corresponding to the acquired gestures of the instructor; inputting a processing intention to process a portion of the first hologram corresponding to the type of work, in response to the operation of the instructor; generating a second hologram obtained by processing the first hologram based on a processing definition that combines information on the portion of the first hologram to be processed and information on the processing method, in response to the input processing intention; and outputting the second hologram to the worker's terminal.

2. The work instruction display device according to claim 1, wherein the processor performs the following processes: inputting video of the work environment from the worker's terminal; displaying the input video of the work environment on a display unit; compositing the first hologram or the second hologram onto the input video of the work environment; displaying the video of the work environment with the first hologram or the second hologram composited onto it on the display unit; and outputting the video of the work environment with the first hologram or the second hologram composited onto it to the worker's terminal and displaying it on the display unit of the worker's terminal.

3. The work instruction display device according to claim 2, wherein both the display unit of the work instruction display device and the display unit of the worker's terminal are the display units of a wearable terminal.

4. A program to cause a processor to function by executing: a process to acquire the gestures of an instructor who instructs an operator to perform a task; a process to generate a first hologram corresponding to the acquired gestures of the instructor; a process to input a processing intention for processing a portion of the first hologram corresponding to the type of work, in response to the operation of the instructor; a process to generate a second hologram obtained by processing the first hologram based on a processing definition that combines information on the portion of the first hologram to be processed and information on the processing method, in response to the input processing intention; and a process to output the second hologram to the operator's terminal.

Images