Object information display control device and program

Abstract

One aspect of the present invention comprises: a first processing unit that generates first object information about an operation target that moves with a first motion delay amount in accordance with an operation of an operator; a second processing unit that generates second object information about the operation target that moves with a second motion delay amount smaller than the first motion delay amount in accordance with the operation of the operator; and a third processing unit that displays the first object information and the second object information within the same field of view of the operator.

Description

Object Information Display Control Device and Program

[0001] One aspect of this invention relates to an object information display control device and program used to control the display of object information in a system that controls an object to be operated, such as a robot or an avatar, in conjunction with the operation of an operator.

[0002] For example, a system has been proposed that allows an operator to control a robot located in a remote location or an avatar in a virtual space in conjunction with their own actions. According to this system, the operator can move the robot located remotely or their own avatar in a virtual space in conjunction with actions such as their own gestures and movements.

[0003] Incidentally, in this type of system, in order to move objects such as robots and avatars smoothly and responsively, it is necessary to minimize the delay between the operator's timing of the object's movement and the operator's timing of the operation. In this regard, for example, Non-Patent Document 1 suggests that if the movement delay is approximately 200 msec or less, the coordination between the operator and the object will increase, and it may be possible to control the object with responsiveness as if it were one's own body.

[0004] Ismail MAF, Shimada S (2016) “'Robot' Hand Illusion under Delayed Visual Feedback: Relationship between the Senses of Ownership and Agency” PLOS ONE 11(7): e0159619. https: / / doi.org / 10.1371 / journal.pone.0159619

[0005] However, the connection between the operator and the robot, or between the operator and the information processing device that generates the virtual space, is usually made via a network. Therefore, depending on the network's transmission bandwidth and the performance of the communication equipment, operational delays can become significant. In this case, the coordination between the operator and the object can be impaired, leading to a decrease in the operability of the object.

[0006] This invention was made in view of the above circumstances, and aims to provide a technology that improves the operability of object information by maintaining a high degree of coordination between the operator and object information even when there is a large operational delay.

[0007] To solve the above problems, one embodiment of the object information display control device according to the present invention comprises: a first processing unit that generates first object information of an object to be operated, which operates with a first operating delay amount in response to the operator's operation; a second processing unit that generates second object information, which operates with a second operating delay amount smaller than the first operating delay amount in response to the operator's operation; and a third processing unit that displays the first object information and the second object information within the same field of view of the operator.

[0008] According to one aspect of this invention, in addition to the information of the first object to be operated on, information of a second object that operates with a smaller operating delay than the first object information is displayed in advance. As a result, the operator perceives the second object information, which has a smaller operating delay, as more reliable, and perceives that the operating delay of the first object information to be operated on approaches the display timing of the operating delay of the second object information.

[0009] Therefore, even if the operational delay of the first object information being manipulated is large due to, for example, transmission delays in the network or processing delays in the information processing device, it is expected that the operator will be able to perform operations on the first object information smoothly without losing coordination.

[0010] In other words, according to one aspect of this invention, it is possible to provide a technology that maintains a high degree of coordination between the operator and object information even when there is a large operational delay, thereby improving the operability of the object information.

[0011] Figure 1 is a diagram showing an example of an object control system according to one embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of an object information display control device according to one embodiment of the present invention. Figure 3 is a block diagram showing an example of the software configuration of an object information display control device according to one embodiment of the present invention. Figure 4 is a diagram used to give an overview of object information display control according to one embodiment of the present invention. Figure 5 is a flowchart showing an example of the processing procedure and processing content of the object information display control process executed by the control unit of the object information display control device shown in Figure 3. Figure 6 is a diagram for explaining an example of an initial calibration process to match the initial posture of an object with the posture of the operator. Figure 7 is a diagram showing an example of calibration information calculated by the initial calibration process. Figure 8 is a diagram showing an example of the operation information acquisition process. Figure 9 is a diagram showing an example of the storage result of the operator's operation information. Figure 10 is a diagram showing an example of the storage result of information about the object being operated on. Figure 11 is a diagram for explaining an example of the object motion conversion process among the object information display control process shown in Figure 5. Figure 12 is a diagram for explaining an example of the object motion display information generation process among the object information display control process shown in Figure 5. Figure 13 shows an example of operation display information generated by the object operation display information generation process. Figure 14A is a diagram illustrating a first example of the display format of operation display information. Figure 14B is a diagram illustrating a second example of the display format of operation display information. Figure 14C is a diagram illustrating a third example of the display format of operation display information.

[0012] Embodiments of this invention will be described below with reference to the drawings.

[0013] [One Embodiment] (Summary) In one embodiment of this invention, in addition to the object to be operated on, a preceding object that has the same movement as the object to be operated on and has a small amount of operational delay is displayed within the same field of view as the object to be operated on.

[0014] Figure 4 is a diagram illustrating the outline of one embodiment of the present invention. For example, when the object to be operated OB1, such as a robot located in a remote location or an avatar of the operator themselves located in a virtual space, is displayed with an operating delay amount ΔT, a preceding object OB2 is generated and displayed that has the same shape and movement as the object to be operated OB1, and moves with an operating delay amount Δt (Δt < ΔT) that is smaller than ΔT.

[0015] When a preceding object OB2, which moves with a relatively small operating delay Δt, is displayed within the same field of view as the target object OB1, the operator perceives the preceding object OB2, which has a smaller operating delay, as more reliable, and perceives the operating delay ΔT of the target object OB1 as approaching the display timing of the operating delay Δt of the preceding object OB2.

[0016] This is based on cue integration theory, one theoretical explanation for intentional binding, where the time between a spontaneous action and its consequences is perceived as compressed. According to cue integration theory, the time of a relatively unreliable event is perceived as moving towards the time of a more reliable event.

[0017] Furthermore, the cue integration theory is described in detail in the following reference: Reference: Wolpe, N., Haggard, P., Siebner, HR et al. “Cue integration and the perception of action in intentional binding”, Exp Brain Res 229, 467-474 (2013), Internet <URL: https: / / doi.org / 10.1007 / s00221-013-3419-2>.

[0018] Therefore, according to one embodiment of this invention that applies the above-mentioned clue integration theory, even if the operating delay amount ΔT of the target object OB1 is large due to the effects of transmission delay in the network or processing delay in the information processing device, the effect of the preceding target object OB2, which is displayed in advance with a smaller operating delay amount Δt, will cause the operator to perceive the operating delay amount ΔT of the target object OB1 as smaller. As a result, it is expected that the operator's operation on the target object OB1 will be able to be performed smoothly without losing the interconnectedness between them.

[0019] (Configuration Example) (1) System diagram 1 is a diagram showing an example of the configuration of an object control system according to one embodiment of the present invention.

[0020] One embodiment of the object control system controls the movements of the operator's own arm avatar, which is the object to be controlled in the virtual space, and the arm avatar of the preceding object, in conjunction with the movement of the operator's upper limbs. The object to be controlled may be, for example, a robot existing in the physical space at a remote location to realize tele-existence. The preceding object may be, for example, a robot existing in the space where the operator is located.

[0021] To implement control according to one embodiment, a motion sensor SS is attached to the operator US, for example, the wrist. The motion sensor SS is equipped with, for example, an acceleration sensor and an angular velocity sensor, and detects the movement trajectory of the operator US's upper limb. The detected movement trajectory is transmitted as operation information to the object information display control device CS via a wireless interface.

[0022] In addition, other motion sensors (SS) can be used, such as optical motion capture using a depth camera. Furthermore, as a wireless interface, it is possible to use an interface employing a low-power wireless digital communication standard such as Bluetooth (registered trademark).

[0023] A system according to one embodiment includes a display device DP for displaying video information of a virtual space generated on, for example, a server computer. The display device DP uses, for example, a liquid crystal or organic EL element and is connected to an object information display control device CS, for example, via a signal cable.

[0024] In addition, other display devices (DPs) may be used, such as video projectors that project virtual space images, head-mounted displays (HMDs), or smart glasses.

[0025] (2) Object Information Display Control Device CS The Object Information Display Control Device CS is configured, for example, by a personal computer. The Object Information Display Control Device CS may also be a mobile information terminal such as a smartphone or tablet, or a server computer located on the Web or in the cloud.

[0026] Figures 2 and 3 are block diagrams showing examples of the hardware and software configurations of the object information display control device CS, respectively.

[0027] The object information display control device CS includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU), and a storage unit having a program storage unit 2 and a data storage unit 3, a sensor interface (hereinafter referred to as I / F) unit 4, and an input / output I / F unit 5 are connected to this control unit 1 via a bus 6.

[0028] The motion sensor SS is connected to the sensor I / F unit 4 via a wireless interface. The sensor I / F unit 4 receives operation information output from the motion sensor SS and converts it into a signal format that the control unit 1 can process.

[0029] The display device DP is connected to the input / output interface (I / F) unit 5. The I / O interface (I / F) unit 5 outputs and displays operation display information, such as the movement of the target object and the preceding object in the virtual space, which is generated by the control unit 1, to the display device DP. Input / output interface (I / F) unit 5 may also be connected to input devices and external memory, which are not shown in the diagram.

[0030] The program storage unit 2, for example, combines a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), as a storage medium, with a non-volatile memory such as ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores application programs necessary to execute various processes according to one embodiment of this invention.

[0031] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD that can be written to and read at any time, and a volatile memory such as RAM (Random Access Memory), and its storage area is provided with an operation information storage unit 31, a calibration information storage unit 32, a target information storage unit 33, and a display pattern storage unit 34.

[0032] The operation information storage unit 31 is used to temporarily store the trajectory of the upper limb movements performed by the operator US to move the object being operated, as operation information of the operator US.

[0033] The calibration information storage unit 32 stores calibration information generated to match the initial position coordinates of the poses of the target object and the preceding object with the initial pose of the operator's (US) upper limb.

[0034] The target information storage unit 33 stores attribute information about each object of the operation target and the preceding object. The attribute information about the object includes, for example, the amount of operation delay, the joint shape or configuration of the object, and the display form.

[0035] The display pattern storage unit 34 stores information representing the display pattern of each object of the operation target and the preceding target. The display pattern storage unit 34 is also used to store information representing the motion of each object of the operation target and the preceding target.

[0036] The control unit 1 includes, as processing function units according to an embodiment of the present invention, an operation information acquisition processing unit 11, an initial calibration processing unit 12, a motion conversion processing unit 13, and a motion display information generation processing unit 14.

[0037] Each of the above processing function units 11 to 14 is realized by causing the hardware processor of the control unit 1 to execute the application program stored in the program storage unit 2. Note that part or all of the above processing function units 11 to 14 may be realized using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0038] When the operator US operates the operation target object in the virtual space, the operation information acquisition processing unit 11 receives the operation information transmitted from the motion sensor SS via the sensor I / F unit 4, and temporarily stores the received operation information in the operation information storage unit 31.

[0039] Prior to the operation of the operation target object by the operator US, the initial calibration processing unit 12 sets the initial position coordinates of the posture of each object of the operation target and the preceding target with respect to the initial posture of the upper limb of the operator US. Then, the initial calibration processing unit 12 stores the set initial position coordinates of the posture of each object of the operation target and the preceding target in the calibration information storage unit 32 in association with the coordinate information representing the initial posture of the upper limb of the operator US.

[0040] The motion conversion processing unit 13 converts the motion trajectory represented by the operation information read from the operation information storage unit 31 into information representing the motion of each object of the operation target and the preceding target (for example, coordinate information in a virtual space) based on the calibration information stored in the calibration information storage unit 32. Then, the motion conversion processing unit 13 saves the converted information representing the motion of each object of the operation target and the preceding target in the display pattern storage unit 34.

[0041] The motion display information generation processing unit 14 refers to the target information related to each object of the operation target and the preceding target stored in the target information storage unit 33, and the display pattern stored in the display pattern storage unit 34, to generate motion display information in which the motions of the respective objects of the operation target and the preceding target are reflected. Then, the motion display information generation processing unit 14 outputs the generated motion display information from the input / output I / F unit 5 to the display device DP for display.

[0042] (Operation Example) Next, an operation example of the object information display control device CS configured as described above will be described in accordance with the processing procedure of the control unit 1.

[0043] FIG. 5 is a flowchart showing an example of the processing procedure and processing content of the object information display control process executed by the control unit 1 of the object information display control device CS.

[0044] (1) Calibration Information Setting When the control unit of the object information display control device CS detects the start of an operation in step S10, under the control of the operation information acquisition processing unit 11 in step S11, the control unit receives operation information representing the motion of the upper limb of the operator US transmitted from the motion sensor SS via the sensor I / F unit 4, and stores the received operation information in the operation information storage unit 31.

[0045] FIG. 6 is a diagram showing an example of a coordinate space used for explaining the calibration information setting process. As shown in FIG. 6, the operation information includes the origin coordinate O of the upper limb of the operator US, the length L of the upper limb, and the initial position coordinate Po of the operator movable part, which is the part that performs the operation instruction.

[0046] In step S12, the control unit of the object information display control device CS determines whether the operation is the first operation based on the operation information. If it is the first operation, in step S13, under the control of the initial calibration processing unit 12, the initial position coordinates of the target object OB1 and OB2 are calculated in order to match the initial posture of the target object OB1 and OB2 with the initial posture of the operator US represented by the operation information, that is, in order to calibrate the target object OB1 and OB2.

[0047] For example, the initial calibration processing unit 12 stores preset values ​​Lt and Ls for the lengths of objects representing the state in which the upper limbs are extended forward, in the calibration information storage unit 32 in advance, in accordance with the initial posture of the operator's upper limbs. Then, for each of the target object OB1 and OB2 of the operation target and the preceding object, the scaling coefficients rt and rs are calculated as follows, using the length L of the operator's upper limbs included in the operation information and its preset values ​​Lt and Ls, respectively: rt = Lt / L rs = Ls / L

[0048] The initial calibration processing unit 12 stores the scaling coefficients rt and rs calculated as described above, the origin coordinates Ot and Os, the initial position coordinates of the operator's movable part, and the preset length values ​​Lt and Ls for each of the target object OB1 and OB2, the target object and the preceding object, in the calibration information storage unit 32, along with the origin coordinates O of the operator's upper limb and the initial position coordinates Po of the operator's movable part, which are included in the operation information.

[0049] Figure 7 shows an example of calibration information stored in the calibration information storage unit 32 after being configured as described above.

[0050] (2) Conversion of operation information to object motion information (2-1) Acquisition of motion trajectory When the calibration processing of the initial position coordinates is completed, the control unit 1 of the object information display control device CS, under the control of the operation information acquisition processing unit 11 in step S14, receives the information representing the movement trajectory of the upper limb of the operator US from the motion sensor SS via the sensor I / F unit 4, and temporarily stores the received information representing the movement trajectory in the operation information storage unit 31 as operation information together with the information representing the operation time.

[0051] For example, it is assumed that the operator US moves the hand, which is a movable part for operation instruction, from position P1 to P2 in the real space as shown in FIG. 8. In this case, the operation information acquisition processing unit 11 obtains the three-dimensional coordinates P1 (x representing the positions P1 and P2 in the real space １ , y １ , z １ ), P2 (x ２ , y ２ , z ２ ) together with their operation times t1 and t2, and stores them in the operation information storage unit 31. When the operator further moves the hand to position P ｎ , similarly, the three-dimensional coordinates Pn (x ｎ , y ｎ , z ｎ ) is acquired together with the operation time tn, and stored in the operation information storage unit 31. FIG. 9 shows an example of the operation information of the operator US stored in the operation information storage unit 31.

[0052] (2-2) Conversion of motion information When the above operation information is acquired, the control unit 1 of the object information display control device CS then, under the control of the motion conversion processing unit 13, first in step S15, reads out the motion trajectory and the operation time from the operation information storage unit 31, and reads out the initial position coordinates Po of the operator US, and the initial position coordinates of each of the operation target and preceding target objects OB1 and OB2 from the calibration information storage unit 32.

[0053] Next, in step S16, the motion conversion processing unit 13 calculates the amount of change from the initial position coordinates P0 of the coordinates P1 and P2 representing the motion trajectory of the operator US, based on the information read out, for each operation time t1, t2, ..., tn. By adding the calculated amount of change to the initial position coordinates of the target object OB1 and OB2 and the preceding object OB1 and OB2, the coordinates of the target object and the preceding object OB1 and OB2 after movement are obtained in the virtual space. In other words, the motion conversion processing unit 13 converts the motion information of the operator US's movable part in real space into motion information of the target object and the preceding object OB1 and OB2 after movement in the virtual space.

[0054] For example, as shown in Figure 11, if the position of the operator US's hand, which is a movable part, moves from coordinate P1 to P2, the motion conversion processing unit 13 converts the virtual space position coordinates of the target object OB1 and OB2, respectively, from Pt1 to Pt2 and Ps1 to Ps2, according to the change.

[0055] More specifically, the motion conversion processing unit 13 offsets the origin coordinates O in real space to the origin coordinates Ot and Os of the target object and the preceding object in virtual space, respectively, and calculates the position coordinates Pt and Ps of the movable parts of each object OB1 and OB2 of the target object and the preceding object using scaling coefficients rt and rs so that the arm lengths become Lt and Ls.

[0056] In step S17, the motion conversion processing unit 13 then determines whether or not objects OB1 and OB2 have joints based on the attribute information of the target of operation and the preceding target stored in the target information storage unit 33.

[0057] For example, suppose that the target information storage unit 33 has predefined attribute information for the target object and the preceding object, as shown in Figure 10. In this case, the motion conversion processing unit 13 determines that there is a joint because the target information storage unit 33 has defined shoulder-to-elbow length and elbow-to-hand length for both the target object and the preceding object OB1, OB2.

[0058] If it is determined that there is a joint, the motion conversion processing unit 13, in step S18, calculates joint coordinates Pt_j and Ps_j for each of the target object OB1 and the preceding target object OB2, using a calculation method defined by inverse kinematics, for example as shown in Figure 12, based on the shoulder-to-elbow length Lt1 and elbow-to-hand length Lt2 of the target object OB1 and the shoulder-to-elbow length Ls1 and elbow-to-hand length Ls2 of the preceding target object OB2, which are stored in the target information storage unit 33.

[0059] The motion conversion processing unit 13 stores the position coordinates Pt, Ps and joint coordinates Pt_j, Ps_j calculated for each of the target object OB1 and OB2, respectively, in the display pattern storage unit 34.

[0060] Figure 13 shows an example of a display pattern storage unit 34 in which the position coordinates Pt, Ps and joint coordinates Pt_j, Ps_j of the movable parts of each object OB1, OB2 are stored together with the display time t and the display pattern.

[0061] (3) Generation and output of operation display information (3-1) Generation of operation display information The control unit 1 of the object information display control device CS then, under the control of the operation display information generation processing unit 14, in step S19, reads out information representing the operation delay amount ΔT, Δt and the display form from the target information storage unit 33 for each of the target object OB1, OB2 and the preceding object, and also reads out the position coordinates Pt, Ps and joint coordinates Pt_j, Ps_j of the movable part and the display pattern from the display pattern storage unit 34.

[0062] Then, in step S20, the operation display information generation processing unit 14 generates operation display information for displaying the target object OB1 and OB2, respectively, on the display device DP as follows.

[0063] In other words, for each of the objects OB1 and OB2, the target object and the preceding object, the operation display processing unit 14 generates operation display information at each operation time t1, t2, ..., tn, setting the display form to the form read from the target information storage unit 33, for example, a robot and its avatar, and setting the display coordinates to the position coordinates Pt, Ps and joint coordinates Pt_j, Ps_j calculated by the operation conversion processing unit 13.

[0064] (3-2) Output of Operation Display Information The operation display information generation processing unit 14 outputs the generated operation display information for each of the above-mentioned objects OB1 and OB2 to the display device DP from the input / output I / F unit 5 in step S21, according to the display pattern read from the display pattern storage unit 34, and at timings in which operation delay amounts ΔT and Δt are added for each of the above-mentioned operation times t1, t2, ..., tn, for display.

[0065] Figures 14A, 14B, and 14C show examples of display patterns. Of these, Figure 14A divides the display area of ​​the display screen of the display device DP into left and right halves, and each divided display area E L , E R This shows the operation display information for each object OB1 and OB2, which are the target of the operation and the preceding target, respectively.

[0066] Figure 14B shows a case where a smaller secondary display area Es is provided within a portion of the main display area of ​​the display screen, and the preceding target object OB2 is displayed in a reduced size within this secondary display area Es. In other words, the display sizes of the target object OB1 and the preceding target object OB2 do not need to be different.

[0067] Figure 14C shows a case where the preceding object OB2, represented by a transparent image, is displayed adjacent to the target object OB1, represented by a real image. In this case, the preceding object OB2, represented by a transparent image, may be displayed superimposed on the target object OB1, represented by a real image.

[0068] Furthermore, the display patterns of the target object OB1 and OB2, as well as the preceding object, can be in any positional relationship as long as they fit within the field of view of the operator (US). For example, the target object OB1 and OB2 may be displayed separately on two adjacent display devices, and the types of display devices may be different. However, the joint shapes and operation details of each object OB1 and OB2 must be identical.

[0069] (4) Operation termination The object information display control device CS monitors whether the operation has ended in step S22. If the operation is continuing, it returns to step S14 to acquire the operation trajectory and then executes the series of display controls described above again in steps S15 to S22. Similarly, as long as the operation is continuing, the display controls are repeatedly executed in steps S14 to S22, and when the operation ends, the display controls also end.

[0070] (Effects) As described above, in one embodiment, in a system that controls an object OB1, such as a robot located in a remote location or an avatar in a virtual space, to be operated by an operator US, to move in accordance with the operator US's movements, when displaying the object OB1 to be operated for the operator US, a preceding object OB2 is generated which has the same joint shape and movement content as the object OB1 to be operated and which is set to have a smaller movement delay than the object OB1 to be operated. The generated preceding object OB2 is then displayed together with the object OB1 to be operated within the same field of view in accordance with the operator US's movements.

[0071] Therefore, even if the operation delay ΔT of the target object OB1 is large, the preceding target object OB2 is displayed in the same field of view with a smaller operation delay Δt. As a result, the operator US perceives the operation delay ΔT of the target object OB1 as smaller, and the operator US can smoothly operate the target object OB1 without losing the coordination between their own actions and the actions of object OB1.

[0072] Furthermore, when calculating the display position coordinates of the target object OB1 and OB2, the system calculates not only the coordinates of the operator's movable parts, such as the hand giving the operation, but also the coordinates of the arm joints. Based on the calculated coordinates of the operator's movable parts and joints, it generates motion display information for the target object OB1 and OB2. As a result, even though the operator's movements are detected by a single motion sensor SS attached to the wrist, it is possible to display not only the hand's movements but also the movements of the arm joints.

[0073] [Other Embodiments] (1) In one embodiment, the case in which the preceding object OB2 is displayed horizontally on the same screen as the object OB1 to be operated on was described as an example. However, the invention is not limited to this, and the preceding object OB2 may be displayed superimposed on the object OB1 to be operated on.

[0074] (2) Each processing unit 11-14 and each storage unit 31-34 of the object information display control device CS does not necessarily have to be provided in a single information processing device, but may be distributed across multiple personal computers or server computers.

[0075] (3) In addition, the configuration of the processing unit, processing procedures and content, types, shapes, sizes, and color schemes of the objects to be operated on and prior objects of the object information display control device can also be modified in various ways without departing from the spirit of this invention.

[0076] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0077] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0078] Furthermore, the functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0079] Furthermore, there is memory for storing computer programs that include computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0080] CS: Object Information Display Control Unit SS: Sensor DP: Display Device 1: Control Unit 2: Program Storage Unit 3: Data Storage Unit 4: Sensor I / F Unit 5: Input / Output I / F Unit 6: Bus 11: Operation Information Acquisition Processing Unit 12: Initial Calibration Processing Unit 13: Motion Conversion Processing Unit 14: Motion Display Information Generation Processing Unit 31: Operation Information Storage Unit 32: Calibration Information Storage Unit 33: Target Information Storage Unit 34: Display Pattern Storage Unit

Claims

1. An object information display control device comprising: a first processing unit that generates first object information of an object to be operated, which operates with a first operating delay amount in response to the operator's operation; a second processing unit that generates second object information of an object to be operated, which operates with a second operating delay amount smaller than the first operating delay amount in response to the operator's operation; and a third processing unit that displays the first object information and the second object information within the same field of view of the operator.

2. The object information display control device according to claim 1, wherein the second processing unit generates second object information that corresponds to the first object information in terms of shape and operation content, and operates with a second operation delay amount smaller than the first operation delay amount, in response to the operation of the operator.

3. The object information display control device according to claim 1, wherein the third processing unit comprises a first display pattern that divides the display screen into a plurality of display areas and displays the first object information and the second object information side by side in each of these display areas, and a second display pattern that displays the first object information represented by a real image and the second object information represented by a transparent image side by side or superimposed, and the first object information and the second object information are displayed by selectively using the first display pattern and the second display pattern.

4. A program that causes a processor in an object information display control device to execute a process performed by a first to third processing unit in an object information display control device that references any one of claims 1 to 3.

Images