Information processing device and method

Abstract

An information processing device according to one embodiment comprises: a generation unit that, on the basis of a gesture indicating a work instruction from an instructor of remote cooperative work to a worker, generates a hologram, which is presented to the worker, according to the content of the work instruction to the worker; a determination unit that, on the basis of biological information about the worker who is presented with the hologram and works, determines whether it is necessary to process the content of the hologram presented to the worker; and a processing unit that, when the determination unit determines that it is necessary to process the hologram presented to the worker, processes the content of the hologram presented to the worker.

Description

Information processing apparatus and method

[0001] Embodiments of the present invention relate to an information processing apparatus and method.

[0002] In recent years, situations where workers perform collaborative work remotely, such as remote work, have been increasing. Even among collaborative work, there is a type of collaborative work (hereinafter referred to as "remote collaborative work") in which a work controller (hereinafter referred to as the "instructor") gives instructions to a worker via a call from a remote location or using on-site video. For example, in troubleshooting and repair work for servers or OA equipment, or troubleshooting and repair work for machinery installed outdoors, a method has been studied in which the worker wears a transparent head-mounted display (HMD) with a built-in camera and shares video with the instructor remotely (see, for example, Non-Patent Document 1).

[0003] Furthermore, in remote collaborative work, there is a method of efficiently conveying work content to the worker by superimposing and displaying hand model information obtained from the instructor in the worker's field of view using an HMD (see, for example, Non-Patent Document 2).

[0004] Shunsuke Ichihara, Yusuke Suzuki, "On-site Evaluation Experiment of Remote Work Support System Using Hand Gesture and Drawing Function," IPSJ Interaction 2017, 1-502-13, pp.154-157, Feb. 2017; Shohei Yamada, Niwala P. Chandrasiri, "Verification of Work Efficiency in Remote Work Support Using Smart Glasses and Hand Gestures," The Journal of the Institute of Image Electronics Engineers of Japan, vol.47, no.4, pp.401-404, April 2018.

[0005] In remote collaborative work, relatively many of the materials, tools and target equipment handled by workers require careful handling, so there are problems shown in the following (1) to (3): (1) In order for the instructor to confirm whether instructions have been conveyed to the worker, the instructor must actively check with the worker. (2) Especially when work instructions are complex, the worker may miss work instructions from the instructor. (3) Recognition discrepancies such as mistakes or errors caused by (1) or (2) described above may lead to the occurrence of work errors and an increase in the time required to confirm work instructions.

[0006] This invention was made in view of the above circumstances, and its purpose is to provide an information processing device and method that enables appropriate remote instructions to be given to workers.

[0007] An information processing device according to one aspect of the present invention includes: a generation unit that generates a hologram to be presented to a worker according to the content of a work instruction given to the worker based on a gesture indicating a work instruction from a remote collaborative work instructor to the worker; a determination unit that determines whether or not it is necessary to process the content of the hologram presented to the worker based on the biometric information of the worker who performs the work after receiving the hologram; and a processing unit that processes the content of the hologram presented to the worker when the determination unit determines that it is necessary to process the content of the hologram presented to the worker.

[0008] An information processing method according to one aspect of the present invention is a method performed by an information processing device, comprising: a generation unit of the information processing device generating a hologram to be presented to a worker according to the content of a work instruction given to the worker, based on a gesture indicating a work instruction from a remote collaborative work instructor to the worker; a determination unit of the information processing device determining whether or not it is necessary to process the content of the hologram presented to the worker, based on the biometric information of the worker who performs the work after receiving the hologram; and a processing unit of the information processing device processing the content of the hologram presented to the worker when the determination unit has determined that it is necessary to process the content of the hologram presented to the worker.

[0009] According to the present invention, appropriate remote instructions can be given to workers.

[0010] Figure 1 is a diagram showing an example of application of an information processing system according to one embodiment of the present invention. Figure 2 is a flowchart showing a first example of processing at the instructor terminal. Figure 3 is a flowchart showing a second example of processing at the instructor terminal. Figure 4 is a flowchart showing an example of first processing at the worker terminal. Figure 5 is a flowchart showing an example of second processing at the worker terminal. Figure 6 is a flowchart showing an example of overall processing of the information processing system according to this embodiment. Figure 7 is a flowchart showing a first example of processing at the instructor terminal in the first application example. Figure 8 is a flowchart showing a second example of processing at the instructor terminal in the first application example. Figure 9 is a flowchart showing an example of first processing at the worker terminal in the first application example. Figure 10 is a flowchart showing an example of second processing at the worker terminal in the first application example. Figure 11 is a flowchart showing a first example of processing at the instructor terminal in the second application example. Figure 12 is a flowchart showing a second example of processing at the instructor terminal in the second application example. Figure 13 is a flowchart showing an example of first processing at the worker terminal in the second application example. Figure 14 is a flowchart showing an example of the second processing at the worker terminal in the second application example. Figure 15 is a block diagram showing an example of the hardware configuration of a worker terminal according to one embodiment of the present invention.

[0011] The following describes embodiments of this invention. In this embodiment, when an instructor gives work instructions to a worker from a remote location, both the worker and the instructor wear wearable terminals. The instructor loads gestures into the instructor terminal based on the video of the work object sent from the worker.

[0012] The instructor's terminal processes the actions or shapes it wants to emphasize to the worker, such as by repeatedly playing or slowing down the video, and re-renders them from the worker's perspective. In addition, the worker's terminal suggests to the instructor how to redisplay the actions or shapes to the worker based on the worker's biological state information. Such suggestions may also be made by devices other than the worker's terminal, such as the instructor's terminal or an external cloud server.

[0013] In this embodiment, it is possible to confirm whether or not work instructions have been communicated from the instructor to the worker without the instructor having to actively check. Furthermore, in this embodiment, even with complex work instructions, the worker can receive the instructions accurately without missing any. Therefore, discrepancies in recognition such as completed mistakes or errors can be suppressed, and a reduction in work errors and a reduction in the time required to confirm work instructions can be achieved.

[0014] Figure 1 shows an example of the application of an information processing system according to one embodiment of the present invention. As shown in Figure 1, the information processing system according to one embodiment of the present invention is a system having a worker terminal 100 and an instructor terminal 200 that are connected to each other in a manner that allows them to communicate with each other. The worker terminal 100 is a terminal device handled by a worker in remote collaborative work at the work site, and the instructor terminal 200 is a terminal device handled by an instructor in remote collaborative work.

[0015] The worker terminal 100 includes a camera 110, a synthesis unit 120, a display unit 130, a worker status acquisition unit 140, and a worker status determination unit 150. The instructor terminal 200 includes a gesture acquisition unit 210, a hologram generation unit 220, a gesture recording unit 230, a processing intent input unit 240, a processing definition DB (database) 250, a highlighting and redisplay creation unit 260, an output unit 270, a synthesis unit 280, and a display unit 290.

[0016] The parts of the worker terminal 100 and the instructor terminal 200 will now be described. The camera 110 acquires the worker's work status and outputs the resulting camera image of the worker to the synthesis unit 120 and the synthesis unit 280 of the instructor terminal 200. The synthesis units 120 and 280 synthesize the camera image with a hologram, such as coordinates. The display unit 130 displays the hologram superimposed on the camera image using a display device built into the worker terminal 100 or an external display device. This presents the worker with a hologram corresponding to the instructions for remote collaborative work. The display unit 290 displays the hologram superimposed on the camera image using a display device built into the instructor terminal 200 or an external display device. This presents the hologram presented to the worker to the instructor as well.

[0017] The worker status acquisition unit 140 acquires the worker's biometric information using, for example, a sensor (not shown) attached to the worker terminal 100. This biometric information may include, for example, heart rate, gaze, or sweating.

[0018] The worker status determination unit 150 determines the worker's status based on information from the worker status acquisition unit 140, and based on this worker's status, proposes to the processing intent input unit 240 of the instruction terminal 200 to redisplay the gesture, and also transmits the proposal to redisplay the gesture to the instructioner via the display unit 290.

[0019] The gesture acquisition unit 210 acquires gesture information indicating gestures made by the user using a camera or sensor (not shown) attached to the user terminal 200, outputs this gesture information to the hologram generation unit 220, and also records it in the internal memory of the gesture recording unit 230. The gesture information is, for example, a set of coordinates of feature points of the gesture. The hologram generation unit 220 generates a hologram from the gesture information. The gesture recording unit 230 records the gesture information acquired by the gesture acquisition unit 210 for a certain period of time.

[0020] The processing intent input unit 240 receives input from the instructor, for example, via an input device (not shown) on the instructor terminal 200, after the instructor has confirmed the suggestion to redisplay the processing intent gesture, which indicates how the instructions should be redisplayed for the operator. This processing intent includes, for example, the start time, end time, and method of processing the hologram for the instruction to be processed. The processing definition DB 250 stores a processing definition, which is a combination of the type of hologram and the processing content of this hologram. This processing content is at least one of the following: for example, repeat playback, slow playback, enlargement, reduction, or color processing of the hologram, displaying an arrow as a new hologram, and displaying a trajectory with a dotted line as a new hologram.

[0021] The highlighting and redisplaying creation unit 260, upon receiving input of processing intent from the processing intent input unit 240, creates a hologram for highlighting and redisplaying by processing the gesture information recorded in the gesture recording unit 230 according to the processing intent and the processing definition stored in the processing definition DB 250. The output unit 270 transmits information about the hologram generated by the hologram generation unit 220 or created by the highlighting and redisplaying creation unit 260, such as coordinates, to the synthesis units 120 and 280.

[0022] Furthermore, the synthesis unit 120, display unit 130, worker status acquisition unit 140, and worker status determination unit 150 of the worker terminal 100, as well as the gesture acquisition unit 210, hologram generation unit 220, gesture recording unit 230, processing intent input unit 240, processing definition DB 250, highlighting and redisplay creation unit 260, output unit 270, synthesis unit 280, and display unit 290 of the instructor terminal 200, may be provided on an information processing device separate from the worker terminal 100 and the instructor terminal 200, for example, on a cloud server that can communicate with the worker terminal 100 and the instructor terminal 200, and the creation of the hologram and other operations described above may be performed by this information processing device exchanging information between the worker terminal 100 and the instructor terminal 200.

[0023] Figure 2 is a flowchart showing a first example of processing at the instruction terminal. This processing is the so-called normal processing, without the emphasis redisplay described above. In this normal processing, prior to processing by the output unit 270, the instruction terminal 200 communicates with the gesture acquisition unit 210, hologram generation unit 220, gesture recording unit 230, and synthesis unit 280, indicated by the symbol a in Figure 1. The gesture acquisition unit 210 of the instruction terminal 200 uses a camera or sensor to acquire gesture information, such as coordinates, that represents the gesture expressing the instructions given by the instructioner to the worker (S11). The hologram generation unit 220 generates a hologram from the gesture information acquired in S11 (S12). The output unit 270 transmits the hologram information, such as coordinates, generated in S12 to the synthesis unit 120 and the 280 of the instruction terminal 200 (S13).

[0024] Furthermore, the gesture recording unit 230 records the gesture information acquired in S11 (S14). The synthesis unit 280 creates a hologram superimposed on the camera image by combining the hologram transmitted in S13 with the camera image from the worker terminal 100 (S15). The display unit 290 displays the hologram superimposed on the camera image, created in S15, on a display device built into or external to the instruction terminal 200 (S16).

[0025] Figure 3 is a flowchart showing a second example of processing at the instruction terminal. This processing is performed when the hologram is redisplayed according to the processing intent described above. In this processing, prior to processing by the output unit 270, the instruction terminal 200 interacts with the gesture recording unit 230, processing intent input unit 240, processing definition DB 250, and highlighting redisplay creation unit 260, as indicated by the symbol b in Figure 1. The processing intent input unit 240 of the instruction terminal 200 receives input of the processing intent specified by the instructioner's operation, in this case the start time, end time, and processing method of the processing target, in response to the proposal for hologram redisplay from the worker terminal 100 (S21). The highlighting redisplay creation unit 260 generates a hologram processed using the specified processing method sequentially from the information of the specified start time to the information of the end time, using the gesture information recorded in S14 (S22).

[0026] The output unit 270 transmits the hologram information generated in S22, such as coordinates, to the synthesis unit 280 and the synthesis unit 120 of the worker terminal 100 (S23). The synthesis unit 280 combines the hologram transmitted in S23 with the camera image from the worker terminal 100 to create a hologram superimposed on the camera image (S24). The display unit 290 displays the hologram superimposed on the camera image, created in S24, on the display device (S25). Even without a suggestion for redisplay from the worker terminal 100, the user can specify the processing intent at any time using the processing intent input unit 240.

[0027] Figure 4 is a flowchart showing an example of the first processing at the worker terminal. The camera 110 of the worker terminal 100 photographs the work environment by the worker, outputs the camera image obtained from this photography to the synthesis unit 120, and also transmits the camera image to the instructor terminal 200 (S31).

[0028] The synthesis unit 120 creates a hologram superimposed on the camera image by combining the hologram information transmitted from the instruction terminal 200 with the camera image output in S31 (S32). The display unit 130 displays the hologram superimposed on the camera image, created in S32, on the display device (S33). Figure 5 is a flowchart showing an example of the second processing at the worker terminal. The worker status acquisition unit 140 of the worker terminal 100 acquires the worker's biological information using sensors, etc. (S41). The worker status determination unit 150 determines whether or not it is necessary to redisplay the hologram based on the worker's biological information acquired in S41, and if it determines that it is necessary, it proposes redisplay to the instruction terminal 200 as shown by the symbol c in Figure 1 (S42).

[0029] Figure 6 is a flowchart illustrating an example of the overall processing of the information processing system according to this embodiment. Figure 6 shows the relationship between the processes S41 and S42 at the worker terminal 100 shown in Figure 5, the processes S21 to S25 at the instructor terminal 200 shown in Figure 3, and the processes S32 and S33 at the worker terminal 100 shown in Figure 4. The camera image handled by S32 shown in Figure 6 is the image obtained by S31 shown in Figure 4.

[0030] In the example shown in Figure 6, after the operations S41 and S42 are performed at the worker terminal 100, the operations S21 to S25 are performed at the instructor terminal 200, followed by the operation S23 at the instructor terminal 200, and then the operations S32 and S33 are performed at the worker terminal 100.

[0031] Next, a specific embodiment will be described. In this embodiment, the display to the worker and the supervisor is provided by a head-mounted display (HMD). The supervisor can view camera footage of the worker's work environment through the HMD they are wearing. The work environment is, for example, the work object and the work location. The work object is, for example, a server machine, and the work location is the place where the work object is placed, for example, on a desk. The camera footage of the work environment may or may not show the worker's hands. Furthermore, the worker's hands do not need to be displayed as a hologram.

[0032] The instructor can also see holograms related to their own gestures through the HMD they wear. The same holograms that the instructor sees, for example, holograms of the same movement and position, can also be seen by the worker through the HMD they wear.

[0033] The worker can view their own work environment through the HMD (Head-Mounted Display) they wear. By placing their hand on this work environment, the worker can see camera footage of their hand in the environment. The worker can also see camera footage in which a hologram of the supervisor is superimposed on the image of the work environment. The position of this hologram corresponds to the position the supervisor wants the worker to see.

[0034] Alternatively, a hologram can be superimposed onto camera footage showing the worker's hands in the work environment. When this hologram moves, the worker can move their hands to keep them from moving away from the hologram. This allows for guidance of the worker's movements.

[0035] Next, the first application example will be explained. The implementation details in this first application example are as follows (A1) to (A9). (A1) The work involves repairing a server failure in a communications building. (A2) A novice worker with little experience in troubleshooting performs troubleshooting work in front of the failed server. (A3) A supervisor, a highly experienced worker with extensive troubleshooting experience, gives work instructions to the worker from an office located far from the work site. (A4) Both the supervisor and the worker wear HMDs. Cameras attached to the HMDs allow them to exchange camera footage of their respective work environments, and audio is exchanged via telephones built into each HMD. (A5) Work instructions from the supervisor are not given via telephone audio, but include hand gestures and pointing using the supervisor's hand model, which are shared on each HMD. (A6) Eye-tracking information, which is biometric information of the worker, is acquired by an eye-tracking sensor built into the worker's HMD. (A7) The instructor receives notification if the worker's gaze is not directed at the instructor's hand model for 15 seconds or more. (A8) The instructor is notified that the worker was not actually looking at the instructor's hand model between 18:00:00 and 18:00:30. (A9) In response to the worker's comment that the instructions were too fast, the instructor replays the instructions from the time period in which the instructions were too fast for the worker in slow motion.

[0036] Figure 7 is a flowchart showing a first example of processing at the instructor terminal in the first application example. This processing is the normal processing without the highlighting and redisplay described above. The gesture acquisition unit 210 of the instructor terminal 200 acquires gesture information, in this case the coordinates of feature quantities such as fingers and joints related to the hand, using sensors built into the HMD worn by the instructor (S111). The hologram generation unit 220 generates a hologram, in this case a hand model, from the gesture information acquired in S111 (S112). The output unit 270 transmits information about the hologram, which is the hand model generated in S112, such as the coordinates, to the synthesis unit 280 and the synthesis unit 120 of the worker terminal 100 (S113).

[0037] Furthermore, the gesture recording unit 230 records the gesture information acquired in S111 (S114). The synthesis unit 280 creates a hologram superimposed on the camera image by combining the hologram information transmitted in S113 with the camera image from the worker terminal 100 (S115). The display unit 290 displays the hologram superimposed on the camera image, created in S115, on the built-in or external display device of the instructor terminal 200 (S116).

[0038] Figure 8 is a flowchart showing a second example of processing at the instruction terminal in the first application example. This processing is performed when the hologram is redisplayed according to the processing intent described above. The processing intent input unit 240 of the instruction terminal 200 receives input of the processing intent specified by the instruction operator after receiving a proposal for redisplaying the hologram from the worker terminal 100 (S121). Here, the proposal for redisplaying the hologram is a proposal to notify on the hologram of "a proposal for redisplaying instructions for the time 18:00:00 to 18:00:30". In this case, the processing intent is the start time and end time of the processing target and the processing method. The start time and end time of the processing target are specified as the time proposed in the processing intent, "18:00:00 to 18:00:30", and the processing method is specified as repeated playback at 0.7 times the speed compared to before processing.

[0039] Prior to specifying this processing method, the instructor confirms with the worker whether they understood the instruction "18:00:00 to 18:00:30". If the worker reports that they did not understand the instruction because it was given too quickly, the processing method is specified by entering the processing intent, including the time period "18:00:00 to 18:00:30" and "0.7x slow playback," into a menu screen that is not shown.

[0040] When the processing intent input unit 240 receives the processing intent input in S121, the enhanced redisplay creation unit 260 converts the gesture information from the instructor recorded in S114 between "18:00:00 and 18:00:30" into a hand model at 0.7 times the speed, according to the processing intent and the processing definition stored in the processing definition DB 250 (S122). The output unit 270 transmits the information of the hand model converted in S122, such as coordinates, to the synthesis unit 280 and the synthesis unit 120 of the worker terminal 100 (S123). The synthesis unit 280 creates a hand model superimposed on the camera image by combining the hand model transmitted in S123 with the camera image from the worker terminal 100 (S124). The display unit 290 displays the hand model superimposed on the camera image, created in S124, on the display device (S125).

[0041] Figure 9 is a flowchart showing an example of the first processing at the worker terminal in the first application example. The camera 110 of the worker terminal 100 photographs the work environment by the worker, outputs the camera image obtained from this photography to the synthesis unit 120, and also transmits the camera image to the instructor terminal 200 (S131).

[0042] The synthesis unit 120 creates a hologram superimposed on the camera image by combining the hologram information transmitted from the instruction terminal 200 with the camera image output in S131 (S132). The display unit 130 displays the hologram superimposed on the camera image, created in S132, on the display device (S133).

[0043] Figure 10 is a flowchart showing an example of the second processing at the worker terminal in the first application example. The worker status acquisition unit 140 of the worker terminal 100 acquires the worker's gaze information using the eye-tracking sensor of the HMD worn by the worker (S141). The worker status determination unit 150, when it is determined from the gaze information acquired in S141 that the worker's gaze has not been directed towards the instructor's hologram for 15 seconds or more, proposes to the instructor terminal 200 to redisplay the hologram for the time when the gaze was not directed (S142).

[0044] In this step S142, it is detected between "18:00:00 and 18:00:30" that the worker's line of sight is not directed within a range of 10 cm from the hand model provided by the instructor as described above, and re-display of the hand model is proposed to the instructor terminal 200.

[0045] Next, a second application example will be described. Implementation details in this second application example are as described in (B1) to (B9) below. (B1) The work is the connection work of optical cables inside a closure of an outdoor utility pole. (B2) A worker who is a beginner with little experience in troubleshooting climbs the utility pole and performs the work. (B3) An instructor who is a skilled expert with rich troubleshooting experience provides work instructions to the worker from an office located far away from the work site. (B4) Both the instructor and the worker wear head-mounted displays (HMDs). Cameras attached to each HMD exchange videos of each other's working environment, and voices are exchanged via telephones built into each HMD. (B5) Work instructions from the instructor are not provided via telephone voice, but include gestures and pointing of the instructor's hand model shared via each HMD. (B6) The amount of perspiration, which is biological information of the worker, is acquired from the exposed skin of the worker by a wearable perspiration sensor, for example, a wristwatch-type or small skin-attached sensor. (B7) The instructor receives a notification to that effect when the worker's perspiration amount increases sharply in the past 3 minutes, for example, when the perspiration amount exceeds a threshold such as when it increases by 50% or more relative to the average perspiration amount. (B8) The instructor is actually notified that the worker's perspiration amount has increased sharply between the time "18:01:00" and "18:01:05". (B9) After receiving a response from the worker via hearing that the worker was anxious about whether he fully understood the series of work instructions, the instructor causes the trajectory of the hand model to be displayed and the hand model to be displayed in an enlarged size for the instructions given during the time period "18:00:00" to "18:01:00", in which the worker felt anxious about the instruction content, and causes the hand model to be re-displayed for the worker. It is known that the amount of perspiration increases due to psychological sweating caused by tension, anxiety, stress or the like, and sensors for measuring the amount of perspiration are commercially available.

[0046] FIG. 11 is a flowchart showing a first example of processing at an instructor terminal in a second application example. This processing is normal-time processing without the above-described enhanced re-display. A gesture acquisition unit 210 of an instructor terminal 200 acquires gesture information, here coordinates of feature amounts such as fingers and joints related to a hand, by means of a sensor built into an HMD worn by the instructor (S211). A hologram generation unit 220 generates a hologram, here a hand model, from the gesture information acquired in S211 (S212). An output unit 270 transmits information of the hologram which is the hand model generated in S212, such as coordinates, to a combining unit 280 and a combining unit 120 of a worker terminal 100 (S213).

[0047] Further, a gesture recording unit 230 records the gesture information acquired in S211 (S214). The combining unit 280 combines the hologram transmitted in S213 with camera video from the worker terminal 100, thereby generating a hologram superimposed on the camera video (S215). A display unit 290 displays the hologram superimposed on the camera video generated in S215 on a built-in or external display device of the instructor terminal 200 (S216).

[0048] FIG. 12 is a flowchart showing a second example of processing at an instructor terminal in a second application example. This processing is processing performed when a hologram is re-displayed according to the above-described processing intention.

[0049] A processing intention input unit 240 of an instructor terminal 200 receives a proposal for re-displaying a hologram from the worker terminal 100, and accepts input of a processing intention specified by an operation of the instructor (S221).

[0050] Here, the proposal for re-displaying a hologram is a proposal notified on the hologram that "proposing re-display of work instructions to be displayed on the display of the HMD, based on the possibility that the worker was tense and anxious during the time period from 18:01:00 to 18:01:05".

[0051] Furthermore, in this case, the processing intent includes the start time, end time, and processing method of the object being processed. In this application example, regarding the tension and anxiety experienced by the worker between "18:01:00 and 18:01:05", the supervisor confirms with the worker whether or not they received work instructions prior to that time. After receiving a response from the worker stating that they were anxious about understanding the work between "18:00:00 and 18:01:00", the supervisor specifies the processing method by entering the processing intent, including the "18:00:00 to 18:01:00 time slot" and "display of the hand model's trajectory," into the menu screen.

[0052] When the processing intent input unit 240 receives the processing intent input in S221, the enhanced redisplay creation unit 260, according to the processing intent and the processing definition stored in the processing definition DB 250, determines the center point of the hand model from the gesture information for the time period "18:00:00 to 18:01:00" recorded in S214, and generates hand model information (S222) in which, in addition to generating a hand model based on the gesture information, the trajectory showing the time transition of the center point of the hand model is represented by a dotted line as a new hologram.

[0053] The output unit 270 transmits the information of the hand model generated in S222 to the synthesis unit 280 and the synthesis unit 120 of the worker terminal 100 (S223). The synthesis unit 280 combines the information of the hand model transmitted in S223 with the camera image from the worker terminal 100 to create a hand model superimposed on the camera image (S224). The display unit 290 displays the hand model superimposed on the camera image, created in S224, on the display device (S225).

[0054] Figure 13 is a flowchart showing an example of the first processing at the worker terminal in the second application example. The camera 110 of the worker terminal 100 photographs the work environment by the worker, outputs the camera image obtained from this photography to the synthesis unit 120, and also transmits the camera image to the instructor terminal 200 (S231).

[0055] The synthesis unit 120 creates a hologram superimposed on the camera image by combining the hologram information transmitted from the instruction terminal 200 with the camera image output in S231 (S232). The display unit 130 displays the hologram superimposed on the camera image, created in S232, on the display device (S233).

[0056] Figure 14 is a flowchart showing an example of the second processing at the worker terminal in the second application example. Here, a wearable sweat sensor is attached to the worker's exposed skin, and the worker status acquisition unit 140 of the worker terminal 100 measures the amount of sweat produced by the worker using the sweat sensor attached to the worker (S241).

[0057] The worker condition determination unit 150 detects when the latest amount of sweat measured in S241 has increased rapidly compared to the amount of sweat the worker has produced over the past 3 minutes, for example, when the increase exceeds a threshold such as 50% of the average amount of sweat produced over the past 3 minutes.

[0058] The worker status determination unit 150 notifies the supervisor that the amount of sweat produced by the worker has increased rapidly, along with information about the time period during which this increase occurred, for example, "18:01:00 to 18:01:05," and suggests to the supervisor terminal 200 that the instructions for the time period preceding this increased period be redisplayed to the worker (S242).

[0059] Figure 15 is a block diagram showing an example of the hardware configuration of a worker terminal according to one embodiment of the present invention. In the example shown in Figure 15, the worker terminal 100 according to the above embodiment is composed of, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115. The worker terminal 100 will be described below, but the same applies to the instructor terminal 200 shown in Figure 1.

[0060] The communication interface 114 includes, for example, one or more wireless communication interface units, enabling the transmission and reception of information with the communication network. As the wireless interface, for example, an interface employing a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0061] The input / output interface 113 is connected to an input device 500 and an output device 600, which are attached to the worker terminal 100 and used by users or the like.

[0062] The input / output interface 113 can take in operation data entered by a user or the like through an input device 500 such as a keyboard, touch panel, or touchpad, and can also output output data to an output device 600, including a display device using liquid crystal or organic EL (Electro Luminescence), for display. The input device 500 and output device 600 may be devices built into the worker terminal 100, or they may be input and output devices of other information terminals that can communicate with the worker terminal 100 via a network.

[0063] The program memory 111B is a non-temporary tangible storage medium in which a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), is used in combination with another non-volatile memory such as ROM (Read Only Memory), and can store programs necessary for executing various control processes, etc., according to one embodiment.

[0064] The data memory 112 is a tangible storage medium that, for example, uses a combination of the above-mentioned non-volatile memory and volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during the process of various operations.

[0065] The worker terminal 100 according to one embodiment of the present invention may be configured as a data processing device having, as a software-based processing function unit, various parts of the worker terminal 100, for example, the synthesis unit 120, display unit 130, worker status acquisition unit 140, and worker status determination unit 150 shown in Figure 1.

[0066] The storage devices used as work memory by each part of the worker terminal 100 may be configured using the data memory 112 shown in Figure 15. However, the storage areas configured by these storage devices are not essential to the worker terminal 100, and may be, for example, external storage media such as USB (Universal Serial Bus) memory, or areas provided in storage devices such as database servers located in the cloud.

[0067] Each processing function in the worker terminal 100 can be implemented by having the hardware processor 111A read and execute a program stored in the program memory 111B. Some or all of these processing functions may be implemented in various other forms, including application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0068] Furthermore, the methods described in each embodiment can be stored as programs (software means) that can be executed by a computer on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memories (ROMs, RAMs, flash memories, etc.), and can also be transmitted and distributed via communication media. The programs stored on the media also include configuration programs that configure the computer to run software means (including not only the execution program but also tables or data structures). The computer implementing this device reads the program recorded on the recording media and, if necessary, constructs the software means using the configuration program, and executes the above-described processes by controlling the operation of this software means. Note that the recording media referred to in this specification are not limited to those for distribution, but also include storage media such as magnetic disks or semiconductor memories provided inside the computer or in devices connected via a network.

[0069] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0070] 100...Worker terminal 200...Instructor terminal 110...Camera 120, 280...Composite unit 130, 290...Display unit 140...Worker status acquisition unit 150...Worker status determination unit 210...Gesture acquisition unit 220...Hologram generation unit 230...Gesture recording unit 240...Processing intent input unit 250...Processing definition DB (database) 260...Highlighted redisplay creation unit 270...Output unit

Claims

1. An information processing device comprising: a generation unit that generates a hologram to be presented to a worker according to the content of a work instruction given to the worker, based on a gesture indicating a work instruction from a remote collaborative work instructor to the worker; a determination unit that determines whether or not it is necessary to process the content of the hologram presented to the worker, based on the biometric information of the worker who performs the work after receiving the hologram; and a processing unit that processes the content of the hologram presented to the worker when the determination unit determines that it is necessary to process the content of the hologram presented to the worker.

2. The information processing apparatus according to claim 1, wherein the processing unit modifies the content of the hologram based on information from the instructor indicating an intention to change the content of the work instruction when the determination unit determines that processing of the hologram presented to the worker is required.

3. The information processing apparatus according to claim 1, wherein when the determination unit determines that processing of the hologram presented to the operator is required, the processing unit processes the content of the hologram presented to the operator by at least one of the following: reducing the speed of the work indicated by the hologram, increasing the number of times the hologram is presented, and adding a new hologram.

4. An information processing method performed by an information processing device, comprising: a generation unit of the information processing device generating a hologram to be presented to the worker according to the content of a work instruction given to the worker, based on a gesture indicating a work instruction from a remote collaborative work instructor to the worker; a determination unit of the information processing device determining whether or not it is necessary to process the content of the hologram presented to the worker, based on the biometric information of the worker who performs the work after receiving the hologram; and a processing unit of the information processing device processing the content of the hologram presented to the worker when the determination unit has determined that it is necessary to process the content of the hologram presented to the worker.

Images