Work instruction display device and computer program

The work instruction display device and program address alignment challenges in remote collaborative work by using holograms that adjust color and provide feedback, enhancing accuracy and efficiency in task performance.

WO2026234751A1PCT designated stage Publication Date: 2026-11-12NT T INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/016776
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-07
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In remote collaborative work, workers have difficulty accurately aligning with instructional hand models due to challenges in visually grasping the position and shape, including depth perception, making it hard to perform precise tasks.

Method used

A work instruction display device and computer program that display holograms as models, adjust their color and transparency based on positional and shape agreement with the worker's hand, and provide feedback through sound and vibration to intuitively guide alignment.

Benefits of technology

Enables workers to accurately align with instructional models by intuitively understanding positional and shape differences, reducing errors and shortening the time to confirm work content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025016776_12112026_PF_FP_ABST
    Figure JP2025016776_12112026_PF_FP_ABST
Patent Text Reader

Abstract

This work instruction display device comprises a processor that executes: processing for displaying, on a display unit, a hologram serving as a model for work; processing for acquiring, in the same coordinate system as the coordinate system in which the hologram is displayed, information pertaining to the position of an object on which a worker performs work that is to be aligned with the hologram; and processing for changing the color of the hologram displayed on the display unit in accordance with the difference between the position at which the hologram is displayed and the acquired position of the object.
Need to check novelty before this filing date? Find Prior Art

Description

Work instruction display device and computer program

[0001] Embodiments of the present invention relate to a work instruction display device and a computer program.

[0002] In recent years, cases where workers collaboratively perform work remotely, such as through telework, have been increasing.

[0003] The collaborative work herein targets collaborative work (hereinafter referred to as "remote collaborative work") in which a supervisor (hereinafter referred to as "instructor") gives instructions to a worker from a remote location using calls and on-site video.

[0004] Examples of remote collaborative work include: ・Example 1: Failure repair work for servers and OA equipment ・Example 2: Failure repair work for outdoor machinery.

[0005] In remote collaborative work, a scheme in which a worker wears a see-through type HMD (Head Mount Display) such as camera-equipped smart glasses to share video with an instructor remotely has been studied.

[0006] In remote collaborative work, a method has been devised in which hand model information acquired from an instructor is superimposed and displayed in the worker's field of view via smart glasses to efficiently convey work details.

[0007] Shunsuke Ichihara, Yusuke Suzuki, "Field Evaluation Experiment of Remote Work Support System Using Hand Gestures and Drawing Functions," IPSJ Interaction 2017, 1-502-13, pp.154-157, Feb. 2017. Shohei Yamada, Naiwala P. Chandrasiri, "Verification of Work Efficiency in Remote Work Support Using Smart Glasses and Hand Gestures," The Journal of the Institute of Image Electronics Engineers of Japan, vol.47, no.4, pp.401-404, April 2018.

[0008] However, the hand model information of the instructor superimposed and displayed in the worker's field of view makes it difficult for the worker to visually accurately grasp the position and shape including the sense of depth, so there is a problem that it is difficult for the worker to perform accurate work according to the instructional instructions that serve as a model.

[0009] This invention has been made in view of the above problems, and aims to provide a work instruction display device and a computer program that enable an operator to intuitively understand the difference between the display position of a model hologram and the physical position of the object to be aligned with the hologram.

[0010] The work instruction display device according to the present invention includes a processor that performs the following: a process of displaying a hologram that serves as a model for the work on a display unit; a process of acquiring information regarding the position of an object on which the worker is to perform work that should be aligned with the hologram, in the same coordinate system as the coordinate system on which the hologram is displayed; and a process of changing the color of the hologram displayed on the display unit according to the difference between the position on which the hologram is displayed and the acquired position of the object.

[0011] According to the work instruction display device of the present invention, it becomes possible for an operator to intuitively understand the difference between the display position of a model hologram and the physical position of the object to be aligned with the hologram.

[0012] Figure 1 is a functional block diagram showing an example of the configuration of a work instruction display system 1 according to the first embodiment of the work instruction display device and computer program of the present invention. Figure 2 is a flowchart showing an example of processing performed by the worker terminal 10 (work instruction display device) of the work instruction display system 1 of the first embodiment. Figure 3 is a functional block diagram showing an example of the configuration of a work instruction display system 1 according to the second embodiment of the work instruction display device and computer program of the present invention. Figure 4 is a flowchart showing an example of processing performed by the worker terminal 10 (work instruction display device) and the instructor terminal 20 of the work instruction display system 1 of the second embodiment. Figure 5 is a block diagram showing an example of the hardware configuration of the work instruction display system 1 according to an embodiment of the present invention.

[0013] Hereinafter, embodiments of the work instruction display device and computer program of the present invention will be described with reference to the drawings.

[0014] (Configuration of the First Embodiment) Figure 1 is a functional block diagram showing an example of the configuration of a work instruction display system 1 according to the first embodiment of the work instruction display device and computer program of the present invention.

[0015] The work instruction display system 1 of the first embodiment includes a worker terminal 10 (work instruction display device) for worker WM who works on the work target Tw according to the instructions of a model stored in advance.

[0016] The worker terminal 10 is realized by having the processor execute the functions of each part shown in Figure 1 according to a computer program (through the cooperation of hardware and software) in an information processing device such as an HMD (Head Mount Display) (wearable device), PC (Personal Computer), or tablet device equipped with a processor (CPU: Central Processing Unit). The worker terminal 10 may be realized by combining multiple types of information processing devices, such as an HMD and a PC, or it may be realized with a single information processing device.

[0017] In the first embodiment, the worker terminal 10 is configured with an HMDw.

[0018] The worker terminal 10 includes a model recording unit 11, a gesture acquisition unit 12, a degree of agreement information creation unit 13, a hologram generation unit 14, a display unit 15, a sound output unit 16, and a vibration generation unit 17.

[0019] The example recording unit 11 records example gesture information. Examples include how to operate a certain device or machine, how to use a certain tool, or how to perform a certain magic trick, where hand movements are important. Gesture information is represented, for example, by a set of coordinates corresponding to the characteristic points of the hand movements (gestures).

[0020] The gesture acquisition unit 12 detects and acquires gesture information (a set of coordinates, etc.) Gew, which includes the hands of the worker WM, the object to be matched to the example, using hand tracking with a camera, sensors, etc.

[0021] The degree of agreement information creation unit 13 compares the example gesture information Gem recorded in the example recording unit 11 with the worker WM's gesture information Gew acquired by the gesture acquisition unit 12 in terms of position and shape, calculates the degree of agreement information, which is the difference between the two, as a quantitative value, and outputs it.

[0022] The degree of agreement information is calculated by comparing the set of coordinates of feature points (Gem:Gew), which are the gesture information of the model and the worker WM. For example, the average distance between the coordinates of the feature points is calculated and this value is used as the positional agreement information, and the variance of the distances between the coordinates of the feature points is calculated and this value is used as the shape agreement information. The set of coordinates of the feature points, which are the gesture information, contains information about the positions of the model and the worker WM, but the combination of adjacent coordinates is included as information that indicates the shape.

[0023] The hologram generation unit 14 generates a hologram (a set of coordinates, etc.) Hg from the gesture information Gew of the example, based on the degree of agreement information (Gem:Gew) between the example and the operator WM, which was created by the degree of agreement information creation unit 13.

[0024] For example, a hologram (hand model) corresponding to the example gesture information Gem is generated, and a hologram Hg is generated by changing the color (gradation, transparency, contrast) of the generated hologram according to the degree of agreement with the operator WM's gesture information Gew.

[0025] For example, when the degree of positional agreement between the model hand and the worker WM's hand (object) is low and far below a preset agreement level, the model hand model Hg is colored red. As the degree of positional agreement increases and the hand gets closer to the preset agreement level, the model hand model Hg is colored closer to green.

[0026] For example, if the degree of shape agreement between the model and the worker WM's hand (object) is lower than a preset agreement level, the transparency of the model's hand model Hg is set to "0%". As the degree of shape agreement approaches the preset agreement level, the transparency of the model's hand model Hg, excluding its outline, is increased.

[0027] Furthermore, the hologram generation unit 14 generates a hologram (hand model) Hg of the model based on the degree of agreement information between the model and the worker WM's hand (object), and may also generate and add holograms such as arrows pointing towards the model hand model Hg to indicate the direction in which the worker WM should correct the position of their hand, corresponding to the difference in position between the model and the worker WM's hand. In addition, the unit may generate and add holograms such as arrows pointing towards the corresponding parts of the model hand model Hg to indicate the direction in which the worker WM should correct the shape of their hand (for example, the degree of finger opening) for each part where the shape differs, corresponding to the difference in shape between the model and the worker WM's hand.

[0028] In this case, the holograms such as arrows may be generated smaller the higher the degree of similarity between the model and the worker's (WM's) hand, and larger the lower the degree of similarity.

[0029] Furthermore, the hologram generation unit 14 generates a hologram (hand model) Hg of the model based on the degree of agreement between the model and the worker WM's hand (object), and may also generate and add holograms of shapes of a size corresponding to the degree of agreement. For example, a circle hologram may be generated as a larger circle the lower the degree of agreement, and as a smaller circle the higher the degree of agreement.

[0030] The display unit 15 overlays a hologram (hand model) Hg of a model, which has been generated by the hologram generation unit 14 by changing the color based on the degree of agreement between the model and the worker WM, onto the image displayed on the HMDw, including the work object Tw and the worker WM's hands (object), along with arrow and graphic holograms corresponding to the degree of agreement.

[0031] The sound output unit 16 outputs, for example, a sound that is louder (or a more noticeable type of sound) as the degree of match is lower, and a sound that is quieter (or a less noticeable type of sound) as the degree of match is higher, based on the degree of match information (Gem:Gew) between the example created by the degree of match information creation unit 13 and the gesture information (Gem:Gew) between the operator WM.

[0032] The vibration generating unit 17 generates vibrations that are larger (or more noticeable) as the degree of agreement between the example created by the degree of agreement information creation unit 13 and the gesture information (Gem:Gew) of the operator WM, for example, as the degree of agreement decreases, and vibrations that are smaller (or less noticeable) as the degree of agreement increases.

[0033] Furthermore, the coordinate systems of the gesture acquisition unit 12 and the display unit 15 of the worker terminal 10 are the same.

[0034] In the work instruction display system 1 of the first embodiment configured in this way, the worker WM can perform the work in accordance with the example by moving their own hand, which is displayed on the display unit 15 (HMDw) of the worker terminal 10, in accordance with the hologram (hand model) Hg corresponding to the example gesture (Gem) displayed on the HMDw, so as not to deviate from the example.

[0035] In this process, the reference hologram (hand model) Hg changes color (gradation, transparency, contrast) based on the degree of agreement with the worker WM, and the higher the degree of agreement, the more the color (gradation, transparency, contrast) that represents the higher the degree of agreement. As a result, the worker WM can intuitively understand the difference between the display position of the reference hologram Hg and the physical position of their own hand (object) that should be aligned with the hologram Hg, and can perform the work accurately according to the reference instructions.

[0036] Furthermore, by adding holograms such as arrows to the example hologram (hand model) Hg to indicate the direction in which the position and shape of the worker WM's hand should be corrected, or by adding holograms of shapes that represent a higher degree of agreement with the worker WM, the worker WM can more intuitively understand and correct the difference between the display position of the example hologram Hg and the physical position of their own hand (object) that should be aligned with the hologram Hg.

[0037] Furthermore, if the system displays a model hologram (hand model) Hg that changes color to indicate a higher degree of matching with the worker WM, and also outputs sounds and vibrations that become louder (more noticeable) as the degree of matching decreases and quieter (less noticeable) as the degree of matching increases, the worker WM can more intuitively understand and correct the difference between the display position of the model hologram Hg and the physical position of their own hand (object) that should be aligned with the hologram Hg.

[0038] In the worker terminal 10 of the first embodiment, the image of the work environment, including the work object Tw and the worker WM's hands (object), displayed on the display unit 15 (HMDw), may be either an image of the work environment as it is using AR (Augmented Reality) or an image of the work environment using a pass-through method with VR (Virtual Reality). In addition, the worker WM's hands (object) may be displayed on the display unit 15 (HMDw) as a hologram (hand model) based on gesture information Gew.

[0039] (Operation of the First Embodiment) Next, the operation of the work instruction display system 1 of the first embodiment will be described.

[0040] Figure 2 is a flowchart showing an example of a process performed by the worker terminal 10 (work instruction display device) of the work instruction display system 1 of the first embodiment.

[0041] Here, the example recording unit 11 pre-records gesture information Gem of the example work by, for example, a content preparer who is an expert, and assumes that, for example, a worker WM who is a trainee wears an HMDw and performs the work by imitating the example hologram Hg displayed on the HMDw.

[0042] In the worker terminal 10, the gesture acquisition unit 12 detects and acquires gesture information (coordinates, etc.) Gew, which includes the worker WM's hand (step S11).

[0043] The matching degree information creation unit 13 compares the model gesture information Gem recorded in the model recording unit 11 and the gesture information Gew of the worker WM acquired by the gesture acquisition unit 12 in terms of position and shape, and calculates and outputs the matching degree information that is the difference therebetween as a quantitative value (step S12).

[0044] The hologram generation unit 14 generates a hologram (hand model) corresponding to the model gesture information Gem, and based on the matching degree information of the gesture information (Gem: Gew) between the model created by the matching degree information creation unit 13 and the worker WM, generates a hologram Hg that changes to a color (gradation, transparency, contrast) expressing that the higher the matching degree is, the higher the matching degree becomes (step S13).

[0045] For example, the gradation is gradually changed from "red" when the matching degree is low to "green" when the matching degree is high. The transparency is gradually changed from the transparency "0%" when the matching degree is low to the transparency "100%" when the matching degree is high. The contrast is gradually changed from the contrast "high" when the matching degree is low to the contrast "low" when the matching degree is high.

[0046] The display unit 15 displays the model hologram Hg generated by the hologram generation unit 15 on the HMDw worn by the worker WM (step S14).

[0047] Here, the hologram generation unit 14 may add and display a hologram such as an arrow for indicating the direction in which the position and shape of the hand of the worker WM should be corrected on the model hologram (hand model) Hg displayed on the display unit 15 (HMDw) based on the matching degree information created by the matching degree information creation unit 13, or may add and display a graphic hologram expressing that the higher the matching degree with the worker WM is, the higher the matching degree becomes.

[0048] Further, the sound output unit 16 may output, based on the matching degree information of the gesture information (Gem: Gew) between the model created by the matching degree information creation unit 13 and the worker WM, for example, a louder sound (or a more noticeable type of sound) as the matching degree is lower, and a quieter sound (or a less noticeable type of sound) as the matching degree is higher.

[0049] Further, the vibration generation unit 17 may generate, based on the matching degree information of the gesture information (Gem: Gew) between the model created by the matching degree information creation unit 13 and the worker WM, for example, a stronger vibration (or a more noticeable type of vibration) as the matching degree is lower, and a weaker vibration (or a less noticeable type of vibration) as the matching degree is higher.

[0050] (Summary of the First Embodiment) As described above, according to the work instruction display system 1 of the first embodiment, the worker WM can perform work in alignment with the model by moving their own hand displayed on the HMDw so as not to deviate, in accordance with the hologram (hand model) Hg corresponding to the model gesture (Gem) projected on the display unit 15 (HMDw) of the worker terminal 10. At this time, the model hologram (hand model) Hg is displayed while changing to a color (gradient, transparency, contrast) representing a higher matching degree as the matching degree increases, based on the matching degree information with the worker WM, so that the worker WM can intuitively understand and correct the difference between the display position of the model hologram Hg and the physical position of their own hand (target object) that should be aligned with the hologram Hg, and can perform accurate work according to the model instruction. Thereby, work errors of the worker WM can be reduced, and the time for the worker WM to confirm the work content can be shortened.

[0051] Therefore, according to the work instruction display system 1 of the first embodiment, the worker can intuitively understand the difference between the display position of the model hologram and the physical position of the target object that should be aligned with the hologram.

[0052] (Configuration of the Second Embodiment) Figure 3 is a functional block diagram showing an example of the configuration of the work instruction display system 1 according to the second embodiment of the work instruction display device and the computer program of the present invention.

[0053] The work instruction display system 1 of the second embodiment includes, for example, an instructioner terminal 20 of an instructioner DM that gives work instructions to a worker WM from a remote location, which is connected via a communication network, and a worker terminal 10 (work instruction display device) of a worker WM that performs work on the work target Tw in response to instructions from the instructioner DM.

[0054] Both the instructor terminal 20 and the worker terminal 10 are implemented using information processing devices such as HMDs (Head Mount Display) (wearable devices), PCs (Personal Computers), and tablet terminals equipped with a processor (CPU: Central Processing Unit), by having the processor execute the functions of each part shown in Figure 3 according to a computer program (through the cooperation of hardware and software). Both the instructor terminal 20 and the worker terminal 10 may be implemented by combining multiple types of information processing devices, such as an HMD and a PC, or by a single information processing device.

[0055] In the second embodiment, the instructor terminal 20 is composed of an HMDi and a PC, and the worker terminal 10 is composed of an HMDw.

[0056] The instruction terminal 20 includes a gesture acquisition unit 21, an output unit 22, a hologram generation unit 23, a synthesis unit 24, and a display unit 25.

[0057] The worker terminal 10 includes a gesture acquisition unit 12, a degree of agreement information creation unit 13, a hologram generation unit 14, a display unit 15, a sound output unit 16, a vibration generation unit 17, and a camera 18.

[0058] In the instructor terminal 20, the gesture acquisition unit 21 uses a camera, sensors, etc., to detect and acquire gesture information (a set of coordinates, etc.) Gei of the example instructor DM's hand.

[0059] The output unit 22 outputs the gesture information Gei of the instructor DM's hand, acquired by the gesture acquisition unit 21, to the hologram generation unit 23, and also outputs and transmits it to the worker terminal 10 via the communication network.

[0060] The hologram generation unit 23 generates a hologram (set of coordinates, etc.) Hgi (hand model) corresponding to the gesture information Gei of the hand of the instructioner DM output by the output unit 22.

[0061] The synthesis unit 24 synthesizes (overlays the hologram on the video) the video Pw of the work environment, including the work target Tw, which is input (transmitted) from the worker terminal 10, with the hologram Hgi (hand model) of the instructor DM's hand, which is generated by the hologram generation unit 23.

[0062] The display unit 25 displays a video Pw of the work environment on the HMDi, which is superimposed with a hologram Hgi (hand model) of the instructor DM's hand. The video Pw of the work environment includes, for example, the work object Tw such as a server machine and the location such as the desk on which the work object Tw is placed, but the hands of the worker WM may or may not be visible. Also, the hologram Hgi of the instructor DM's hand may or may not be visible.

[0063] The instructor (DM) will demonstrate the task on the target Twitter account by watching a video (Pw) of the work environment, including the target Twitter account, and demonstrating the task.

[0064] In the worker terminal 10, the gesture acquisition unit 12 detects and acquires gesture information (a set of coordinates, etc.) Gew, which includes the hands of the worker WM, the object that should be matched to the hand movements of the instructor DM, the model, using hand tracking with a camera, sensors, etc.

[0065] The degree of agreement information creation unit 13 compares the gesture information Gei of the example instructor DM's hand transmitted from the instructor terminal 20 with the gesture information Gew of the worker WM acquired by the gesture acquisition unit 12 in terms of position and shape, calculates the degree of agreement information, which is the difference between the two, as a quantitative value, and outputs it.

[0066] The degree of agreement information is calculated by comparing the set of coordinates of feature points (Gei:Gew), which are the gesture information of the instructor DM and the worker WM. For example, the average distance between the coordinates of the feature points is calculated and this value is used as the positional agreement information, and the variance of the distances between the coordinates of the feature points is calculated and this value is used as the shape agreement information. The set of coordinates of the feature points, which are the gesture information, is information about the positions of the instructor DM and the worker WM, but the combination of adjacent coordinates is included as information that indicates the shape.

[0067] The hologram generation unit 14 generates a hologram (a set of coordinates, etc.) Hgd from the gesture information Gei of the instructor DM, based on the degree of agreement information (Gei:Gew) between the instructor DM and the worker WM created by the degree of agreement information creation unit 13.

[0068] For example, a hologram (hand model) corresponding to the gesture information Gei of the instructor DM is generated, and a hologram Hgd is generated by changing the color (gradation, transparency, contrast) of the generated hologram according to the degree of agreement with the gesture information Gew of the worker WM.

[0069] For example, when the degree of positional agreement between the instructor DM's hand and the worker WM's hand (object) is low and far below a preset degree of agreement, the instructor DM's hand model Hgd is colored red. As the degree of positional agreement increases and the hands get closer to the hand, the instructor DM's hand model Hgd is colored closer to green.

[0070] For example, if the degree of shape agreement between the instructor DM's hand and the worker WM's hand (object) is lower than a predetermined agreement level, the transparency of the instructor DM's hand model Hgd is set to "0%". As the degree of shape agreement approaches the predetermined agreement level, the transparency of the hand model Hgd other than the outline is increased.

[0071] Furthermore, the hologram generation unit 14 generates a hologram (hand model) Hgd of the instructor DM's hand based on the degree of agreement information between the instructor DM's hand and the worker WM's hand (object). It may also generate and add holograms such as arrows pointing towards the direction of the instructor DM's hand model Hgd to indicate the direction in which the worker WM should correct the position of their hand, corresponding to the difference in position between the instructor DM's hand and the worker WM's hand. Additionally, it may generate and add holograms such as arrows pointing towards the corresponding parts of the instructor DM's hand model Hgd to indicate the direction in which the worker WM should correct the shape of their hand (e.g., the degree of finger spread) for each part where the shape differs, corresponding to the difference in shape between the instructor DM's hand and the worker WM's hand.

[0072] In this case, the hologram, such as an arrow, may be generated smaller the higher the degree of similarity between the hand of the instructor (DM) and the hand of the worker (WM), and larger the lower the degree of similarity.

[0073] Furthermore, the hologram generation unit 14 generates a hologram (hand model) Hgd of the instructor DM's hand based on the degree of agreement between the instructor DM's hand and the worker WM's hand (object), and may also generate and add holograms of shapes of a size corresponding to the degree of agreement. For example, a circle hologram may be generated as a larger circle for lower degrees of agreement and as a smaller circle for higher degrees of agreement.

[0074] The display unit 15 superimposes a hologram (hand model) Hgd of the instructor DM, which is generated by the hologram generation unit 14 by changing the color based on the degree of agreement between the instructor DM and the worker WM, onto the image displayed on the HMDw, including the work object Tw and the worker WM's hands (object), along with arrows and geometric holograms corresponding to the degree of agreement.

[0075] The sound output unit 16 outputs, for example, a sound that is louder (or more noticeable) as the degree of agreement between the instructor DM and the worker WM, based on the degree of agreement information (Gei:Gew) created by the degree of agreement information creation unit 13, and a sound that is quieter (or less noticeable) as the degree of agreement increases.

[0076] The vibration generating unit 17 generates vibrations that are larger (or more noticeable) as the degree of agreement between the gesture information (Gei:Gew) between the instructor DM and the worker WM, created by the degree of agreement information creation unit 13. For example, vibrations that are smaller (or less noticeable) as the degree of agreement increases.

[0077] Camera 18 acquires video (image) Pw of the work environment, including the worker WM's work target Tw, and transmits it to the instructor terminal 20 via the communication network.

[0078] Furthermore, the coordinate systems of the gesture acquisition unit 12 and display unit 15 of the worker terminal 10 and the gesture acquisition unit 21 and display unit 25 of the instructor terminal 20 are the same. The image of the work environment displayed on the worker WM's HMDw with the hologram Hgd corresponding to the instructor DM's gesture information Gei superimposed, and the image of the work environment Pw displayed on the instructor DM's HMDi with the hologram Hgi corresponding to the instructor DM's gesture information Gei superimposed, are the same except for changes in color or the addition of arrows or geometric holograms to the hologram Hgd based on the degree of agreement information on the worker WM side.

[0079] In the work instruction display system 1 of the second embodiment configured in this way, the worker WM can perform work in accordance with the hands of the instructor DM, which serve as an example, by moving their own hands, which are displayed on the HMDw, in accordance with the hologram (hand model) Hgd corresponding to the gesture (Gei) of the instructor DM displayed on the display unit 15 (HMDw) of the worker terminal 10, so as not to deviate from the original hand.

[0080] In this process, the hologram (hand model) Hgd of the instructor DM changes color (gradation, transparency, contrast) based on the degree of agreement with the worker WM, and the higher the degree of agreement, the more the color (gradation, transparency, contrast) that represents the higher the degree of agreement. As a result, the worker WM can intuitively understand the difference between the display position of the instructor DM's hologram Hgd and the physical position of their own hand (object) that should be aligned with the hologram Hgd, and can perform the work accurately according to the instructor DM's instructions.

[0081] Furthermore, by adding holograms such as arrows to the hologram (hand model) Hgd of the instructor DM to indicate the direction in which the position and shape of the worker WM's hand should be corrected, or by adding holograms of shapes that represent a higher degree of agreement with the worker WM, the worker WM can more intuitively understand and correct the difference between the display position of the instructor DM's hologram Hgd and the physical position of their own hand (object) that should be aligned with the hologram Hgd.

[0082] Furthermore, if the hologram (hand model) Hgd of the instructor DM changes color to indicate a higher degree of matching with the worker WM, and if sounds and vibrations are generated and output that become louder (more noticeable) as the degree of matching decreases and quieter (less noticeable) as the degree of matching increases, the worker WM can more intuitively understand and correct the difference between the display position of the instructor DM's hologram Hgd and the physical position of their own hand (object) that should be aligned with the hologram Hgd.

[0083] In the worker terminal 10 of the second embodiment, the image of the work environment, including the work object Tw and the worker WM's hands (object), displayed on the display unit 15 (HMDw), may be either an augmented reality (AR) representation of the work environment as is, or a pass-through representation using virtual reality (VR). Furthermore, the worker WM's hands (object) may be displayed on the display unit 15 (HMDw) as a hologram (hand model) based on gesture information Gew.

[0084] (Operation of the second embodiment) Next, the operation of the work instruction display system 1 of the second embodiment will be described.

[0085] Figure 4 is a flowchart showing an example of the processing performed by the worker terminal 10 (work instruction display device) and the instructor terminal 20 of the work instruction display system 1 of the second embodiment.

[0086] Here, we assume that worker WM is an unskilled worker with little experience working on the target object Tw, and supervisor DM is a skilled worker with extensive experience working on the target object Tw.

[0087] First, at the worker terminal 10, the camera 18 acquires video Pw of the work environment, including the worker WM's work target Tw (step S111). The video Pw of the work environment acquired by the camera 18 is output (transmitted) to the instructor terminal 20 (step S112).

[0088] The gesture acquisition unit 12 starts acquiring gesture information Gew, which includes the hands of the worker WM (step S113).

[0089] The worker (WM) performs tasks on the target device (Tw) while viewing a video of the work environment, including the target device (Tw), displayed on the HMDw.

[0090] First, when the instructor terminal 20 receives the video Pw of the worker WM's work environment, which is output (transmitted) from the worker terminal 10, the synthesis unit 24 outputs the input video Pw of the work environment as is to the display unit 25, and the display unit 25 displays the video Pw of the work environment output from the synthesis unit 24 on the HMDi (step S21).

[0091] The gesture acquisition unit 21 starts acquiring gesture information Gei corresponding to the hand movements of the instructor DM (step S22).

[0092] The instructor DM, while viewing the video Pw of the work environment, including the worker's hands working on the target Tw, displayed on the HMDi, moves their own hands within the gesture acquisition range of the gesture acquisition unit 21's camera and sensors in order to guide the worker WM's work (hand movements).

[0093] When the gesture acquisition unit 21 acquires gesture information Gei corresponding to the hand movements of the instructor DM (step S23 (Yes)), the output unit 22 outputs the acquired gesture information Gei of the instructor DM to the hologram generation unit 23 and also outputs and transmits it to the worker terminal 10 (step S24).

[0094] The hologram generation unit 23 generates a hologram Hgi (hand model) corresponding to the gesture information Gei of the instructioner DM's hand output by the output unit 22 (step S25).

[0095] The synthesis unit 24 superimposes the hologram Hgi, which corresponds to the gesture information Gei of the instructor DM generated by the hologram generation unit 23, onto the video Pw of the work environment input from the worker terminal 10 and displayed on the HMDi (step S26).

[0096] The display unit 25 causes the HMDi to display the video Pw of the work environment, which has been synthesized with the hologram Hgi (step S27).

[0097] The instructor (DM) guides the hand movements of the worker (WM) while superimposing a hologram (Hgi) corresponding to the worker's hand movements onto the video (Pw) of the work environment, including the worker's hands, displayed on the HMDi.

[0098] When gesture information Gei corresponding to the hand movements of the instructor DM, output (transmitted) from the instructor terminal 20, is input to the worker terminal 10 (step S114 (Yes)), the degree of agreement information creation unit 13 compares the gesture information Gei of the example instructor DM, input from the instructor terminal 20, with the gesture information Gew of the worker WM, acquired by the gesture acquisition unit 12, in terms of position and shape, and calculates the degree of agreement information, which is the difference between them, as a quantitative value and outputs it (step S115).

[0099] The hologram generation unit 14 generates a hologram (hand model) corresponding to the gesture information Gei of the instructor DM, and generates a hologram Hgd that changes in color (gradation, transparency, contrast) to express a higher degree of agreement as the degree of agreement increases, based on the degree of agreement information (Gei:Gew) between the instructor DM and the worker WM created by the degree of agreement information creation unit 13 (step S116).

[0100] For example, the gradient gradually changes from "red" when the degree of matching is low to "green" when the degree of matching is high. Similarly, the transparency gradually changes from "0%" when the degree of matching is low to "100%" when the degree of matching is high. Furthermore, the contrast gradually changes from "strong" when the degree of matching is low to "weak" when the degree of matching is high.

[0101] The display unit 15 displays the hologram Hgd of the instructor DM, generated by the hologram generation unit 14, on the HMDw worn by the worker WM (step S117).

[0102] Here, the hologram generation unit 14 may add holograms such as arrows to indicate the direction in which the position and shape of the worker WM's hand should be corrected, based on the degree of agreement information created by the degree of agreement information creation unit 13, to the example hologram (hand model) Hgd of the instructor DM displayed on the display unit 15 (HMDw), or it may add and display holograms of shapes that represent a higher degree of agreement as the degree of agreement with the worker WM increases.

[0103] Furthermore, the sound output unit 16 may output, for example, a sound that is louder (or more noticeable) as the degree of agreement between the instructor DM and the worker WM, based on the degree of agreement information (Gei:Gew) created by the degree of agreement information creation unit 13, and a sound that is quieter (or less noticeable) as the degree of agreement is higher.

[0104] Furthermore, the vibration generating unit 17 may generate vibrations that are larger (or more noticeable) as the degree of agreement between the gesture information (Gei:Gew) between the instructor DM and the worker WM, created by the degree of agreement information creation unit 13, for example, vibrations that are smaller (or less noticeable) as the degree of agreement increases.

[0105] (Example 1 of the second embodiment) Assume that when a server device in a communications building malfunctions, a novice with little experience in troubleshooting (worker WM) receives instructions from a highly experienced worker (instructor DM) located in a remote office and performs the task of turning off the power to the server device (target Tw).

[0106] The worker WM and the supervisor DM each wear an HMDw and an HMDi, share video Pw of the work environment acquired by the camera 18 attached to the HMDw, and communicate via voice using the telephones built into the HMDw and HMDi (steps S111, S112 → S21).

[0107] Here, based on the relative coordinates of the instructor DM's HMDi and their hand, the instructor DM's hand hologram (hand model) Hgd is displayed on the worker WM's HMDw. For example, if the instructor DM's hand is located at [5 cm forward, 3 cm to the right, 8 cm down] from the HMDi, then the instructor DM's hand model Hgd will be displayed at the same position [5 cm forward, 3 cm to the right, 8 cm down] from the worker WM's HMDw.

[0108] If we take the position of the worker WM's HMDw as the origin, the power button of the server device (work target Tw) is located at [20 cm forward, 5 cm to the right, and 3 cm down].

[0109] Initially, worker WM's hand is positioned [5 cm in front, 1 cm to the right, and 3 cm below] the HMDw.

[0110] The instructor (DM) moves their hand to the location of the server device's power button [20 cm forward, 5 cm to the right, 3 cm down] included in the video Pw of the work environment displayed on the HMDi, points to the server device's power button, and instructs the user to turn off the power.

[0111] The instructor terminal 20 displays a hologram (hand model) Hgi of the instructor DM's hand at the position [20 cm in front, 5 cm to the right, and 3 cm below] in the video Pw of the work environment displayed on the HMDi (steps S21 to S27).

[0112] The instructor (DM) confirms that their own hologram (hand model) Hgi is correctly displayed in the video Pw of the work environment shown on the HMDi, at the location of the server device's power button, and also confirms the status of the worker (WM)'s work on the server device.

[0113] The worker terminal 10 calculates the difference between the gesture information Gei of the instructor DM and the gesture information Gew of the worker WM [15 cm forward, 4 cm to the right, 6 cm up] as the degree of agreement information (steps S113 to S115).

[0114] Based on the degree of agreement information [15 cm forward, 4 cm to the right, 6 cm up], the worker terminal 10 generates a hologram (hand model) Hgd in "red" that corresponds to the gesture information of the instructor DM, and displays it at the position [20 cm forward, 5 cm to the right, 3 cm down] in the video of the work environment visible through the HDMw (steps S116, S117).

[0115] Based on the degree of agreement between the gesture information Gei of the instructor DM and the gesture information Gew of the worker WM, the worker terminal 10 changes the display of the instructor DM's hologram (hand model) Hgd, which is currently displayed in red, to yellow when the distance between the instructor DM's hand model Hgd and the worker WM's hand is within 10 cm, and changes to green when the distance is within 3 cm (steps S113 to S117).

[0116] Even a novice worker (WM) with little experience in troubleshooting can easily and correctly shut down the server device by intuitively understanding that their hand is approaching the hand model (Hgd) of the experienced supervisor (DM) through the color change of the Hgd.

[0117] (Example 2 of the second embodiment) Assume that in indoor and outdoor fault repair work, a novice with little experience in fault response (worker WM) receives instructions on how to use specialized fault repair tools from an experienced worker (instructor DM) located in a remote office.

[0118] The worker WM and the supervisor DM each wear an HMDw and an HMDi, share video Pw of the work environment acquired by the camera 18 attached to the HMDw, and communicate via voice using the telephones built into the HMDw and HMDi (steps S111, S112 → S21).

[0119] Here, based on the relative coordinates of the instructor DM's HMDi and their hand, the instructor DM's hand hologram (hand model) Hgd is displayed on the worker WM's HMDw. For example, if the instructor DM's hand is located at [5 cm forward, 3 cm to the right, 8 cm down] from the HMDi, then the instructor DM's hand model Hgd will be displayed at the same position [5 cm forward, 3 cm to the right, 8 cm down] from the worker WM's HMDw.

[0120] The instructor (DM) provides instructions on how to use the work tools, specifically regarding hand positioning. Since the hand position is to be slightly offset from the tool, accuracy of hand positioning is not a concern.

[0121] Initially, worker WM's hands are in a clenched fist position with all fingers closed.

[0122] The instructor terminal 20 acquires the coordinate information of each feature point corresponding to the shape of the instructor DM's hand and its changes as gesture information Gei by hand tracking, and transmits it to the worker terminal 10 (steps S22 to S24).

[0123] The instructor terminal 20 displays a hologram (hand model) Hgi of the instructor DM's hand, which is generated in accordance with the instructor DM's gesture information Gei, on the video Pw of the work environment displayed on the HMDi (steps S25 to S27).

[0124] The instructor (DM) checks the video Pw of the work environment displayed on the HMDi to confirm that the instructions for using the work tools are being given correctly, and also checks the condition of the worker (WM)'s hand shape.

[0125] The worker terminal 10 generates a hologram (hand model) Hgd corresponding to the gesture information Gei of the supervisor DM, based on the difference between the gesture information Gei of the hand shape used by the supervisor DM when using a work tool and the gesture information Gew of the worker WM's current hand shape (fist), with "0% transparency" and displays it in the video of the work environment visible through HDMw (steps S113 to S117).

[0126] Based on the degree of agreement between the gesture information Gei of the instructor DM and the gesture information Gew of the worker WM, the worker terminal 10 generates and displays the hologram (hand model) Hgd of the instructor DM, which is currently displayed with "0% transparency," so that as the shape of the worker WM's hand approaches the shape of the hand model Hgd of the instructor DM, the outline is removed and the transparency increases up to "100%" (steps S113 to S117).

[0127] Even a novice worker (WM) with little experience in troubleshooting can easily learn the correct way to use the tools by intuitively understanding that their own hand shape is approaching that of the expert supervisor (DM)'s hand model (Hgd) as the transparency changes.

[0128] (Summary of the Second Embodiment) As described above, according to the work instruction display system 1 of the second embodiment, the worker WM can perform work in accordance with the movements of the instructor DM by moving their own hand, which is displayed on the HMDw, in accordance with the hologram (hand model) Hgd corresponding to the gesture (Gei) of the instructor DM, which is displayed on the display unit 15 (HMDw) of the worker terminal 10, so as not to deviate from the HMDw. At this time, the hologram (hand model) Hgd of the instructor DM is displayed in a color (gradation, transparency, contrast) that expresses a higher degree of agreement, based on the degree of agreement information with the worker WM. As a result, the worker WM can intuitively understand and correct the difference between the display position of the hologram Hgd of the instructor DM, which is the model, and the physical position of their own hand (object) that should be aligned with the hologram Hgd, and perform work accurately according to the instructions of the instructor DM. This reduces errors by the worker (WM) and shortens the time the worker (WM) needs to confirm the work content in remote collaborative work where the worker (WM) performs the work according to the example given by the instructor (DM) (Hgd).

[0129] Therefore, according to the work instruction display system 1 of the second embodiment, the worker can intuitively understand the difference between the display position of the example hologram and the physical position of the object to be aligned with the hologram.

[0130] Figure 5 is a block diagram showing an example of the hardware configuration of the work instruction display system 1 according to an embodiment of the present invention.

[0131] In the example shown in Figure 5, the worker terminal 10 (work instruction display device) according to the above embodiment is composed of, for example, a personal computer, an HMD (head-mounted display), or a combination thereof, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0132] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network (network) NW. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0133] The input / output interface 113 is connected to an input device 200 and an output device 300 that are attached to the worker terminal 10 and used by the user (worker WM), etc.

[0134] The input / output interface 113 receives operation data entered by a user or the like through an input device 200 such as a keyboard, touch panel, touchpad, or mouse, and outputs output data to an output device 300, including a display device using liquid crystal or organic EL (electroluminescence), for display. The input device 200 and output device 300 may be devices built into the worker terminal 10 (work instruction display device), or they may be input and output devices of other information processing devices (information processing terminals) that can communicate with the worker terminal 10 via a network NW.

[0135] The program memory 111B is a non-temporary tangible storage medium in which a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), is used in combination with another non-volatile memory such as ROM (Read Only Memory), and stores the programs necessary to execute various control processes according to the embodiment.

[0136] The data memory 112 is a tangible storage medium that, for example, uses a combination of the aforementioned non-volatile memory and volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and generated during the process of various operations.

[0137] An embodiment of the present invention of the worker terminal 10 (work instruction display device) may be configured as a data processing device having, as a software processing function unit, at least a gesture acquisition unit 12, a degree of agreement information creation unit 13, a hologram generation unit 14, a display unit 15, a sound output unit 16, and a vibration generation unit 17 as shown in Figure 1 or Figure 3.

[0138] Each information storage unit (which may include the example recording unit 11) used as a working memory by each part of the worker terminal 10 can be configured using the data memory 112 shown in Figure 5. However, these storage areas are not essential to the worker terminal 10, and may be, for example, areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or in a storage device such as a database server located in the cloud.

[0139] The processing functions in the gesture acquisition unit 12, the degree of agreement information creation unit 13, the hologram generation unit 14, the display unit 15, the sound output unit 16, and the vibration generation unit 17 can all be implemented by having the hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0140] Furthermore, the instruction terminal 20 according to the second embodiment described above is composed of, for example, a server computer, a personal computer (PC), an HMD, or a combination thereof, and, like the worker terminal 10, has a hardware processor (111A) such as a CPU. And, similar to the worker terminal 10, a program memory (111B), a data memory (112), an input / output interface (113), and a communication interface (114) are connected to this hardware processor (111A) via a bus (115).

[0141] The instruction terminal 20 can be configured as a data processing device having at least the gesture acquisition unit 21, output unit 22, hologram generation unit 23, synthesis unit 24, and display unit 25 shown in Figure 3, as a software-based processing function unit.

[0142] The processing functions in the gesture acquisition unit 21, output unit 22, hologram generation unit 23, synthesis unit 24, and display unit 25 can all be implemented, similar to the operator terminal 10, by having a hardware processor (111A) read and execute a program stored in program memory (111B).

[0143] In each of the embodiments described above, the gesture information (Gem) (Gei) used as a model, acquired by the worker terminal 10 (work instruction display device), is not limited to being acquired from the model recording unit 11 or the instruction terminal 20, but may also be acquired from an autonomous device using, for example, artificial intelligence. In this case, the autonomous device may observe the work status of the work target Tw and the worker WM based on the video Pw of the work environment transmitted from the worker terminal 10, generate the gesture information (Gei) used as a model, and output it to the worker terminal 10.

[0144] Furthermore, in each of the embodiments described above, the degree of agreement information creation unit 13 is not limited to a form in which it obtains degree of agreement information based on the difference between the example gesture information (Gem) (Gei) and the worker WM's gesture information Gew. For example, it may obtain degree of agreement information based on the difference between the example hologram (Hg) (Hgi) generated based on the example gesture information (Gem) (Gei) and the worker WM's hologram generated based on the worker WM's gesture information Gew.

[0145] The methods described in each of the embodiments above can be stored as programs (software means) that can be executed by a computer, for example, on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memory, etc.), and can also be transmitted and distributed via communication media.

[0146] Furthermore, the program stored on the medium includes a configuration program that configures the computer with software means (including not only the executable program but also tables and data structures) to be executed by the computer. The computer that implements this device reads the program recorded on the recording medium and, if necessary, constructs the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. In this specification, the recording medium is not limited to distribution media, but also includes storage media such as magnetic disks and semiconductor memory installed inside the computer or in devices connected via a network.

[0147] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the embodiments described above include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed.

[0148] For example, if the problem can be solved and the desired effect can be obtained even if some of the constituent elements shown in the embodiment are removed, then the configuration with these removed constituent elements can be extracted as an invention.

[0149] 1...Work Instruction Display System WM...Worker Tw...Work Object HMDw...Worker's Head-Mounted Display 10...Worker Terminal (Work Instruction Display Device) 11...Example Recording Unit 12...Gesture Acquisition Unit 13...Conformity Information Creation Unit 14...Hologram Generation Unit 15...Display Unit 16...Sound Output Unit 17...Vibration Generation Unit DM...Instructor HMDi...Instructor's Head-Mounted Display PC...Personal Computer 20...Instructor Terminal 21...Gesture Acquisition Unit 22...Output Unit 23...Hologram Generation Unit 24...Synthesis Unit 25...Display Unit

Claims

1. A work instruction display device comprising a processor that performs the following steps: a process of displaying a hologram on a display unit as a model for the work; a process of acquiring information regarding the position of an object on which the worker is to perform work that should be aligned with the hologram, in the same coordinate system as the coordinate system on which the hologram is displayed; and a process of changing the color of the hologram displayed on the display unit according to the difference between the position on which the hologram is displayed and the acquired position of the object.

2. The work instruction display device according to claim 1, wherein the processor performs a process to output a sound to the sound output unit that changes according to the difference between the position where the hologram is displayed and the position of the acquired object.

3. The work instruction display device according to claim 1 or 2, wherein the processor performs a process to generate vibrations in the vibration generating unit that change according to the difference between the position where the hologram is displayed and the position of the acquired object.

4. A computer program that causes a processor to perform the following operations: a process of displaying a hologram on a display unit as a model for a task; a process of acquiring information about the position of an object on which an operator should perform a task, in the same coordinate system as the coordinate system on which the hologram is displayed, in accordance with the coordinate system on which the operator should align the object with the hologram; and a process of changing the color of the hologram displayed on the display unit according to the difference between the position on which the hologram is displayed and the acquired position of the object.