Information processing device and method for controlling virtual input device

Abstract

According to the present invention, a processor provided in an information processing device: includes a step for analyzing a captured image generated by capturing, by means of an outer camera, an image of the periphery of the information processing device, and specifying a movement or the shape of a body feature part of a user of the information processing device; refers to part-key allocation data indicating the correspondence relationship between the movement or the shape of the body feature part created in advance and a key of a virtual input device allocated to the movement or the shape of the body feature part; and receives an input operation of the key of the virtual input device allocated to the movement or the shape of the body feature part.

Description

Information Processing Apparatus and Control Method for Virtual Input Apparatus

[0001] The present invention relates to an information processing apparatus and a control method for a virtual input apparatus.

[0002] Patent Document 1 discloses the following (abstract excerpt): "The display unit is formed of a CRT, a liquid crystal display or the like, and displays a virtual keyboard. The touch panel is a touch panel formed of a substantially transparent material provided on the front surface of the display unit. A control unit displays the virtual keyboard on the display unit, and accepts character input from a user in accordance with a touched position on the touch panel and a display state of the display unit. When the user touches the touch panel, the control unit identifies the touched character from the touch position and the line to which the character belongs, and changes the key arrangement such that an input character display area is located immediately above the line to which the input key belongs, so as to minimize the movement of the user's line of sight."

[0003] Japanese Unexamined Patent Application Publication No. 2006-031518

[0004] When a user wears a head mount display (hereinafter referred to as "HMD") and performs character input using a virtual keyboard as disclosed in Patent Document 1, unlike a physical keyboard, the virtual keyboard does not provide a tactile sensation during key input. Accordingly, a user cannot sense whether key input (including touch operation and gesture operation) to the virtual keyboard has been correctly performed as a sensation at a fingertip, and can only confirm that the key input has been performed after seeing that a character has been added to the character input field. As described above, the conventional character input method using a virtual keyboard has a problem in that input efficiency is low.

[0005] The present invention has been made to solve the above problem, and an object of the present invention is to support input in a virtual space without using a physically existing input apparatus.

[0006] To solve the above problems, the present invention has the configuration described in the claims. To give an example, the present invention is an information processing device comprising a processor, an out-camera, and a depth sensor, wherein the processor analyzes the image generated by the out-camera imaging the area around the information processing device to identify a selected body feature area among the body feature areas of the user of the information processing device, compares a first distance to the body feature area detected by the depth sensor with a second distance to an indicator that touches or taps the body feature area to determine whether the indicator has touched or tapped the body feature area, and if it is determined that the indicator has touched or tapped the body feature area, it refers to pre-created body key assignment data showing the correspondence between the body feature area and the key of a virtual input device assigned to that body feature area, and accepts the input operation of the key of the virtual input device assigned to the body feature area that the indicator has touched or tapped.

[0007] According to the present invention, input in a virtual space can be supported without using an existing input device. Other problems, configurations, and effects will be clarified by the following description of the embodiments.

[0008] This is a hardware configuration diagram of a head-mounted information display device that implements the information processing device according to this embodiment. This is a top view showing the user wearing the information processing device. This is a front view showing the user wearing the information processing device. This is a left side view showing the user wearing the information processing device. This is a functional block diagram of the head-mounted information display device. This is a diagram showing the names of the joints of the user's left hand. This is a diagram showing the names of the parts of the user's left hand. This is a diagram showing the keys assigned to the thumb and palm print. This is a diagram showing the keys assigned to each finger except the thumb. This is a diagram showing the keys displayed using augmented reality objects. This is an explanatory diagram showing input operation example 1. This is an explanatory diagram showing input operation example 2. This is an explanatory diagram showing input operation example 3. This is an explanatory diagram showing input operation example 3. This is an explanatory diagram showing input operation example 4. This is an explanatory diagram showing input operation example 5. This is an explanatory diagram showing input operation example 5. This is an explanatory diagram showing input operation example 6. This is a flowchart showing the flow of the control method of the virtual input device executed by the head-mounted information display device according to this embodiment. This is a diagram showing an example of a Korean keyboard layout. This is a diagram showing an example of a Korean keyboard layout. This is a diagram showing an example of a Korean keyboard layout. This is an explanatory diagram showing an example of an input operation in handwriting mode. This is an explanatory diagram showing an example of an input operation in handwriting mode.

[0009] The information processing device and virtual input device control method according to this embodiment can support input operations of a virtual input device placed in a virtual space by wearing the head-mounted information display device on the head, thereby improving the convenience of the head-mounted information display device. Therefore, since the present invention can enhance the commercial value of the head-mounted information display device to which the present invention is applied, it is expected to contribute to Sustainable Development Goal 8.2 (Increase economic productivity through diversification, technological advancement and innovation, particularly in industries that enhance the value of goods and services and in labor-intensive industries).

[0010] Embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are used throughout the drawings to omit redundant explanations of identical components.

[0011] Figure 1 is a hardware configuration diagram of a head-mounted information display device 100 that incorporates the information processing device according to this embodiment.

[0012] The head-mounted information display device 100 includes a processor 101 using a CPU (Central Processing Unit) and an MCU (Micro Controller Unit), RAM 103 (Random Access Memory), storage 110, a user operation interface (I / F) 120, an image processing unit 130, an audio processing unit 140, a positioning sensor 150, a sensor group 160, a communication interface (I / F) 170, and an expansion I / F 180, all of which are connected to each other by a bus 102.

[0013] Bus 102 is the transmission and reception path for commands and data.

[0014] RAM 103 functions as a work area and primary data storage area for the processor 101.

[0015] The storage 110 is configured using non-volatile storage media such as Flash ROM, EEPROM, SSD, and HDD.

[0016] The user operation interface 120 includes, for example, operation keys 121 and touch sensors 122.

[0017] The operation keys 121 include, for example, switches, a power key, volume keys, etc.

[0018] The touch sensor 122 is configured, for example, as a touchpad and is used to operate a head-mounted information display device 100 or the like. If a separate controller (not shown) is used, the touch sensor 122 may not be necessary.

[0019] The image processing unit 130 includes a display 131, an image signal processing unit 132, an out-camera 133, and an in-camera 134.

[0020] The display 131 may be a transparent display or an opaque display.

[0021] The image signal processing unit 132 is configured, for example, using an image (video) signal processor.

[0022] The rear camera 133 is a camera for capturing peripheral images.

[0023] The front camera 134 is a camera for eye-tracking detection.

[0024] The audio processing unit 140 includes an audio output unit 141, an audio signal processing unit 142, and a microphone 143.

[0025] The audio output unit 141 may include, for example, a speaker or an audio output terminal.

[0026] The audio signal processing unit 142 is configured, for example, using an audio signal processor. The audio signal processing unit 142 may also include an analog-to-digital converter that converts the audio analog signal from the microphone 143 into an audio digital signal, and a digital-to-analog converter that converts the audio digital signal generated by the processor 101 into an audio analog signal.

[0027] Microphone 143 collects ambient sounds and converts them into analog signals.

[0028] The positioning sensor 150 is a sensor that detects the current position of the head-mounted information display device 100, and may be implemented by, for example, a GNSS sensor (GNSS: Global Navigation Satellite System) or a GPS sensor (GPS: Global Positioning System). Alternatively, the method by which the head-mounted information display device 100 acquires position information may be a so-called VPS system that extracts feature points in space by analyzing surrounding images captured by the out-camera 133 and acquires relative position information based on these feature points, instead of using the positioning sensor 150. In this case, the positioning sensor 150 is unnecessary.

[0029] The sensor group 160 includes a gyro sensor 161, a geomagnetic sensor 162, an acceleration sensor 163, and a depth sensor 164. The depth sensor 164 is a sensor that detects the distance from the head-mounted information display device 100 to the object to be measured in depth.

[0030] Communication I / F 170 includes, for example, LAN I / F 171 (LAN: Local Area Network) and BLUETOOTH I / F 172 (BLUETOOTH is a registered trademark).

[0031] The expansion I / F180 is, for example, a USB device connection terminal, used for data transmission and reception, charging, etc.

[0032] Although not shown in the diagram, a vibrator may also be provided. The vibrator is a notification device for the user wearing the head-mounted information display device 100, and may be configured to vibrate under the control of the processor 101 to notify the user, for example, that a specific key has been selected on the virtual keyboard.

[0033] Figure 2A is a top view showing user U1 wearing the head-mounted information display device 100, Figure 2B is a front view showing user U1 wearing the head-mounted information display device 100, and Figure 2C is a left side view showing user U1 wearing the head-mounted information display device 100.

[0034] As shown in Figure 2A, user U1 wears the head-mounted information display device 100 in front of their eyes. In this state, as shown in Figure 2B, the left display 131L is positioned inside the housing of the head-mounted information display device 100, in a position visible to user U1's left eye. The right display 131R is also positioned inside the housing of the head-mounted information display device 100, in a position visible to user U1's right eye. In this embodiment, the left display 131L and the right display 131R may be collectively referred to as the display 131. As an alternative configuration example, a single display 131 may be configured to divide its display area and display images for the left eye and the right eye side by side.

[0035] Furthermore, a left out-camera 133L and a right out-camera 133R are positioned on the outside of the housing of the head-mounted information display device 100. These two cameras are sometimes collectively referred to as the out-camera 133. These two cameras are used to photograph objects around the user U1, that is, around the head-mounted information display device 100.

[0036] Furthermore, a left depth sensor 164L and a right depth sensor 164R are positioned on the outside of the housing of the head-mounted information display device 100. These two depth sensors are sometimes collectively referred to as depth sensor 164. The left depth sensor 164L and the right depth sensor 164R are used to measure the distance to objects around the head-mounted information display device 100. Alternatively, the images captured by the left out-camera 133L and the right out-camera 133R may be used to photograph and measure the distance to objects around the user U1, i.e., around the head-mounted information display device 100. In this case, depth sensor 164 is not necessary. As another example, only one depth sensor 164 may be provided midway between the left out-camera 133L and the right out-camera 133R.

[0037] As shown in Figure 2C, the left side of the head-mounted information display device 100 housing is equipped with a left touch sensor 122L and a left stereo speaker 141L. Although not shown, the right side of the head-mounted information display device 100 is equipped with a right stereo speaker, a monaural microphone, and a charging and wired communication terminal, which is one of the expansion I / F 180s.

[0038] A right touch sensor may be provided on the right side of the head-mounted information display device 100 instead of the left touch sensor 122L. The touch sensor may be installed on either the left or right side, or on both sides.

[0039] Figure 3 is a functional block diagram of the head-mounted information display device 100.

[0040] The storage 110 of the head-mounted information display device 100 includes a basic operation program 1001 that includes basic programs such as an OS, an application program 1002 for realizing the functions of the head-mounted information display device 100 according to this embodiment, and various data storage areas 1009 that serve as areas for storing various operation setting values ​​and various information (video, still images, audio, etc.).

[0041] The basic operation program 1001 and application program 1002 stored in these storages 110 are expanded (loaded) into the RAM 103, and the processor 101 executes the expanded basic operation program 1001 and application program 1002, whereby the functional units and other functional units of the information processing apparatus according to the present embodiment are configured.

[0042] The RAM 103 includes a basic operation functional unit 1101, an application control unit 1102, an image acquisition unit 1111, an input operation determination unit 1112, a display control unit 1113, and a temporary storage area 1199.

[0043] The basic operation functional unit 1101 controls basic operations of the head-mounted information display device 100.

[0044] The application control unit 1102 controls the operation of an application selected and activated by a user U1.

[0045] The image acquisition unit 1111 acquires captured images of the left hand U1HL and right hand U1HR of the user U1 from an image captured and acquired by an out-camera 133.

[0046] The input operation determination unit 1112 analyzes the image captured and acquired by the image acquisition unit 1111 to detect movements of the left hand U1HL and right hand U1HR of the user U1, and determines an input character based on the detection result.

[0047] The display control unit 1113 displays an application image generated based on the control of the application control unit 1102. Furthermore, the display control unit 1113 controls the display 131 such that the input character determined by the input operation determination unit 1112 is displayed in a character input field.

[0048] The temporary storage area 1199 is a temporary storage area for various types of information created / acquired by an application.

[0049] In this embodiment, the head-mounted information display device 100 assigns virtual keyboard keys to characteristic body parts of the user, and has a function that allows the user to input the assigned keys by touching (making contact with the characteristic body part and maintaining that state for a predetermined time or longer) or tapping (making contact with the characteristic body part and releasing it after a short time) the characteristic body part. To realize this function, the head-mounted information display device 100 prepares "body part key assignment data" that associates body parts with keys in advance and stores it in the storage 110.

[0050] In this embodiment, the control method for the virtual input device executed by the head-mounted information display device 100 will be described using an example where keys are assigned to the user's left hand area. However, if the user is left-handed, keys may be assigned to the user's right hand area.

[0051] First, referring to Figures 4A and 4B, we will explain the names of the parts of the left hand as examples of the user's physical characteristics. Figure 4A is a diagram showing the names of the joints of the user's left hand. Figure 4B is a diagram showing the names of the parts of the user's left hand.

[0052] As shown in Figure 4B, the left hand U1HL of user U1 comprises the thenar eminence F0B1, the base of the four fingers F0B3, and the hypothenar eminence F0B5, which are collectively referred to as the palmar print area (thenar eminence F0B1 + base of the four fingers F0B3 + hypothenar eminence F0B5).

[0053] Furthermore, as shown in Figures 4A and 4B, in the thumb (first finger) LF1 of the left hand U1HL, the proximal phalanx F1J1, proximal phalanx F1B1, distal phalanx F1J3, and distal phalanx F1B3 are arranged in order from the palm print towards the fingertip.

[0054] In the left hand, at the index finger (second finger) LF2, the proximal phalanx F2J1, proximal part F2B1, middle phalanx F2J2, middle part F2B2, distal phalanx F2J3, and distal phalanx F2B3 are aligned.

[0055] In the left hand, at the middle finger (third finger) LF3, the proximal phalanx F3J1, proximal phalanx F3B1, middle phalanx F3J2, middle phalanx F3B2, distal phalanx F3J3, and distal phalanx F3B3 are aligned.

[0056] In the ring finger (fourth finger) LF4 of the left hand U1HL, the proximal phalanx F4J1, proximal part of the ring finger F4B1, middle phalanx F4J2, middle part of the ring finger F4B2, distal phalanx F4J3, and distal phalanx F4B3 are aligned.

[0057] In the left hand, at the little finger (fifth finger) LF5, the following points are aligned: proximal phalanx F5J1, proximal part of the little finger F5B1, middle phalanx F5J2, middle part of the little finger F5B2, distal phalanx F5J3, and distal part of the little finger F5B3.

[0058] Figures 5A and 5B show the body part key assignment data in a table structure. The data structure of the body part key assignment data is not limited to a table structure; any data that associates each body part of the left hand with the key assigned to it is acceptable. Figure 5A shows the keys assigned to the thumb and palm print, and Figure 5B shows the keys assigned to each finger excluding the thumb.

[0059] When a user taps a specific area and then taps the same area again within a predetermined time, the input characters will switch in the order shown in the parentheses below. For example, in Figure 5B, the 'GHI4' key is assigned to the middle finger phalanx in English mode. Therefore, when the middle finger phalanx is tapped, 'g' is displayed in the character display field, and when the middle finger phalanx is tapped again within the predetermined time, the character display field changes from 'g' to 'h'.

[0060] Furthermore, if the user taps a specific area and then taps the same area again after a predetermined time has elapsed, the same character will be entered consecutively. For example, in Figure 5B, tapping the proximal phalanx of the middle finger displays "g" in the character display field, and if the proximal phalanx of the middle finger is tapped again after a predetermined time has elapsed, the character display field will change from "g" to "gg".

[0061] In this embodiment, as shown in Figure 5A, the following keys are assigned to the thumb and palm print areas: • Thumb proximal phalanx F1B1: Switches input mode (each tap switches between English letters ⇒ numbers ⇒ symbols ⇒ Japanese ⇒ English letters ⇒…). The selected mode may be highlighted. • Thumb distal phalanx F1B3: Deletes the previous character (backspace). • Thenar eminence F0B1: Moves the cursor position to the left. • Hypothenar eminence F0B5: Moves the cursor position to the right.

[0062] In this embodiment, as shown in Figure 5B, the following keys are assigned to the parts of each finger excluding the thumb. (Proximal phalanx of the index finger F2B1) ・English mode: Input '@-_ / ' and '1' (@⇒-⇒_⇒ / ⇒1⇒@⇒…). ・Number mode: Input '1'. ・Japanese mode: Input characters from the 'a' row (a⇒i⇒u⇒e⇒o⇒a⇒i⇒u⇒e⇒o⇒a⇒…). (Middle phalanx of the index finger F2B2) ・English mode: Input 'ABC' and '2' (a⇒b⇒c⇒A⇒B⇒C⇒2⇒a⇒…). ・Number mode: Input '2'. ・Japanese mode: Input characters from the 'ka' row (ka⇒ki⇒ku⇒ke⇒ko⇒ka⇒…). (End phalanges of the index finger F2B3) ・English mode: Enter 'DEF' and '3' (d⇒e⇒f⇒D⇒E⇒F⇒3⇒d⇒…). ・Number mode: Enter '3'. ・Japanese mode: Enter characters from the 'sa' row (sa⇒shi⇒su⇒se⇒so⇒sa⇒…). (Proximal phalanges of the middle finger F3B1) ・English mode: Enter 'GHI' and '4' (g⇒h⇒i⇒G⇒H⇒I⇒4⇒g⇒…). ・Number mode: Enter '4'. ・Japanese mode: Enter characters from the 'ta' row (ta⇒chi⇒tsu⇒te⇒to⇒tsu⇒ta⇒…). (Middle phalanges of the middle finger F3B2) ・English mode: Enter 'JKL' and '5' (j⇒k⇒l⇒J⇒K⇒L⇒5⇒j⇒…). ・Number mode: Enter '5'.・Japanese mode: Enter characters from the 'na' row (na⇒ni⇒nu⇒ne⇒no⇒na⇒…). (F3B3, distal phalanx of middle finger) ・English mode: Enter 'MNO' and '6' (m⇒n⇒o⇒M⇒N⇒O⇒6⇒m⇒…). ・Number mode: Enter '6'. ・Japanese mode: Enter characters from the 'ha' row (ha⇒hi⇒fu⇒he⇒ho⇒ha⇒…). (F4B1, proximal phalanx of ring finger) ・English mode: Enter 'PQRS' and '7' (p⇒q⇒r⇒s⇒P⇒Q⇒R⇒S⇒7⇒p⇒…). ・Number mode: Enter '7'. ・Japanese mode: Enter characters from the 'ma' row (ma⇒mi⇒mu⇒me⇒mo⇒ma⇒…). (F4B2, middle phalanx of ring finger) ・English mode: Enter 'TUV' and '8' (t⇒u⇒v⇒T⇒U⇒V⇒8⇒t⇒…). - Number mode: Enter '8'. - Japanese mode: Enter characters from the 'ya' row (ya⇒yu⇒yo⇒ya⇒yu⇒yo⇒ya⇒...). (F4B3 at the end of the ring finger) - English mode: Enter 'WXYZ' and '9' (w⇒x⇒y⇒z⇒W⇒X⇒Y⇒Z⇒9⇒w⇒...).- Number mode: Enter '9'. - Japanese mode: Enter characters from the 'ra' row (ra⇒ri⇒ru⇒re⇒ro⇒ra⇒...). (F5B1 on the base of the little finger) - English mode: Convert uppercase letters to lowercase.・ Number mode: Enter '()[]' ((⇒)⇒[⇒]⇒(⇒…). ・ Japanese mode: Add '゛゜' to the preceding character (converts to voiced / semi-voiced consonants) (only for corresponding characters). Or convert uppercase to lowercase (only for corresponding characters). (Middle phalanx of little finger F5B2) ・ English mode: Enter '";:' and '0' ('⇒"⇒;⇒:⇒0⇒'⇒…). ・ Number mode: Enter '0'. ・ Japanese mode: Enter characters from the 'wa' row (wa⇒wo⇒n⇒wa⇒-(long vowel)⇒~⇒wa⇒…). (Last phalanx of little finger F5B3) ・ English mode: Enter ' (space)' and ',.?!' ((space)⇒,⇒.⇒?⇒!⇒(space)⇒…). ・ Number mode: Enter ' (space)' and ',.- / ' ((space)⇒,⇒.⇒-⇒ / ⇒(space)⇒…). - Japanese mode: If there are no unconfirmed characters, enter " (space)" and ". ,? ! " ( (space) ⇒ . ⇒ , ⇒ ? ⇒ ! ⇒ (space) ⇒…). If there are unconfirmed characters, the conversion process for the unconfirmed characters will be performed. - In symbol mode, you may assign 1 to 5 symbols to each part (however, illustration / explanation is omitted).

[0063] Augmented reality objects of keys assigned to specific body parts may be displayed. That is, by overlaying the keys assigned to body parts, users can perform key operations even if they do not remember the key layout corresponding to the body parts. Figure 5C shows a state in which keys are displayed using augmented reality objects. It is also not necessary to overlay the keys assigned to body parts. In particular, if the key assignments to body parts are pre-configured to be the same as the key layout of a 10-key keyboard displayed on an information device such as a smartphone that the user normally uses, the user can easily guess which key is assigned to which body part even without the overlaying of the keys assigned to the body parts, enabling efficient key operation.

[0064] As shown in Figure 5C, key objects may be displayed overlaid on each physical feature. Figure 5C shows an example of key object display in English mode. Note that each finger has clearly defined parts (proximal, middle, and distal phalanges) at the joints, making it easy to understand the relationship between each physical feature and each key, so key objects do not need to be displayed. However, it is preferable to continue displaying the object that represents the current input mode.

[0065] In Figure 5C, the following keys are assigned to parts of each finger, and AR (Augmented Reality) objects are superimposed on those body feature areas. (Palm print of user U1's left hand U1HL) - AR object OBJ01 is displayed on the thenar eminence. - AR object OBJ05 is displayed on the hypothenar eminence. (Thumb (first finger) LF1 of user U1's left hand U1HL) - AR object OBJ11 is displayed on the proximal phalanx of the thumb. - AR object OBJ13 is displayed on the distal phalanx of the thumb. (Index finger (second finger) LF2 of user U1's left hand U1HL) - AR object OBJ21 is displayed on the proximal phalanx of the index finger. - AR object OBJ22 is displayed on the middle phalanx of the index finger. - AR object OBJ23 is displayed on the distal phalanx of the index finger. (User U1's left hand U1HL, middle finger (third finger) LF3) - Display AR object OBJ31 at the proximal phalanx of the middle finger. - Display AR object OBJ32 at the middle phalanx of the middle finger. - Display AR object OBJ33 at the distal phalanx of the middle finger. (User U1's left hand U1HL, ring finger (fourth finger) LF4) - Display AR object OBJ41 at the proximal phalanx of the ring finger. - Display AR object OBJ42 at the middle phalanx of the ring finger. - Display AR object OBJ43 at the distal phalanx of the ring finger. (User U1's left hand U1HL, little finger (fifth finger) LF5) - Display AR object OBJ51 at the proximal phalanx of the little finger. - Display AR object OBJ52 at the middle phalanx of the little finger. - Display AR object OBJ53 at the distal phalanx of the little finger.

[0066] (Input Operation Example 1) Input Operation Example 1 is an example in which the key layout assigned to the body feature area is hidden, and input operations are performed according to the number of times the body feature area is tapped. Figure 6 is an explanatory diagram showing Input Operation Example 1.

[0067] If the user remembers the keys assigned to specific body parts, key input can be performed even without displaying the key layout, by the head-mounted information display device 100 detecting that the user has tapped the body part corresponding to the desired key. Therefore, it is not essential to display the assigned keys on the body parts.

[0068] When user U1 taps various parts of the left hand U1HL with the index finger RF2 of the right hand U1HR, the head-mounted information display device 100 inputs characters corresponding to the keys assigned to each part of the left hand U1HL. The finger used for tapping does not have to be the index finger RF2 of the right hand U1HR. While the index finger RF2 is the finger that is easiest to use for tapping, if, for example, the fingertip is injured, input can be performed with another finger, thus improving the usability of the virtual input device. Alternatively, the user may tap various parts of the left hand U1HL with an operating device (indicator) such as a stylus held in the right hand U1HR.

[0069] In the example in Figure 6, the index finger RF2 of the right hand U1HR taps the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL, so one of the characters 'g', 'h', 'i', 'G', 'H', 'I', or '4' is entered.

[0070] In Figure 6, the key layout that has not been tapped is hidden. However, while the head-mounted information display device 100 performs object recognition processing on the camera's captured image and detects the proximal phalanx F1B1 of the left hand U1HL, the AR object OBJ11 indicating the "symbol aA1" key for switching input modes may be continuously superimposed on the proximal phalanx F1B1 of the left hand U1HL. This makes it easier to confirm the input mode.

[0071] Alternatively, the AR object OBJ11 could be hidden by default, similar to other key layouts, and only displayed when the left hand U1HL thumb proximal phalanx F1B1 is tapped, allowing for confirmation of the input mode.

[0072] In Figure 6, the index finger RF2 of the right hand U1HR taps the proximal phalanges F3B1 of the middle finger LF3 of the left hand U1HL, so one of the characters 'g', 'h', 'i', 'G', 'H', 'I', or '4' is entered. However, it is assumed that English mode is selected by the AR object OBJ11. That is, after tapping the proximal phalanges F3B1 of the middle finger LF3 of the left hand U1HL with the index finger RF2 of the right hand U1HR, 'g' is entered. 'h' is entered if tapped twice in a row within the specified time. 'i' is entered if tapped three times in a row within the specified time. 'G' is entered if tapped four times in a row within the specified time. 'H' is entered if tapped five times in a row within the specified time. 'I' is entered if tapped six times in a row within the specified time. - If you tap seven times in a row within a specified time, the character '4' will be entered. - If you tap eight times in a row within a specified time, the character 'g' will be entered. - The input is confirmed after a specified time has elapsed since the tap. - Alternatively, you can confirm the input character without waiting for the specified time to elapse by tapping a physical feature area that has not been assigned a key (for example, the base of the four fingers F0B3). This is equivalent to assigning a confirmation key to the base of the four fingers F0B3.

[0073] Alternatively, a tap operation on various parts of the left hand U1HL by the index finger RF2 of the right hand U1HR of user U1 may be determined by detecting the distance to the index finger RF2 of the right hand U1HR and the distance to each part of the left hand U1HL using the depth sensor 164, and determining that a tap operation has been performed when the difference between these distances becomes '0' (i.e., contact has occurred).

[0074] Alternatively, a tap operation on various parts of the left hand U1HL by the index finger RF2 of the right hand U1HR of user U1 may be determined to have occurred when the bending or trembling of the left hand U1HL is detected when the various parts of the left hand U1HL are pressed by the index finger RF2 of the right hand U1HR of user U1.

[0075] (Input Operation Example 2) Input Operation Example 2 is an example in which, when user U1 taps a body feature area, an AR object for confirming the input character is displayed on the tapped body feature area. Figure 7 is an explanatory diagram illustrating Input Operation Example 2.

[0076] In Figure 7, when user U1 taps the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL with the index finger RF2 of the right hand U1HR, a confirmation object OBJ31A is displayed at the position of the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL. More specifically: - If the same area is tapped only once within a predetermined time, 'g' is displayed as the confirmation object OBJ31A. - If the same area is tapped twice in a row within a predetermined time, 'h' is displayed as the confirmation object OBJ31A. - If the same area is tapped three times in a row within a predetermined time, 'i' is displayed as the confirmation object OBJ31A. - If the same area is tapped four times in a row within a predetermined time, 'G' is displayed as the confirmation object OBJ31A. - If the same area is tapped five times in a row within a predetermined time, 'H' is displayed as the confirmation object OBJ31A. - If you tap the same area six times in a row within a specified time, the letter 'I' will be displayed as the confirmation object OBJ31A. - If you tap the same area seven times in a row within a specified time, the letter '4' will be displayed as the confirmation object OBJ31A. - If you tap the same area eight times in a row within a specified time, the letter 'g' will be displayed as the confirmation object OBJ31A. - The input will be confirmed after a specified time has elapsed since the tap. At that time, the display shape (color, etc.) of the confirmation object OBJ31A may be changed before and after the input is confirmed.

[0077] (Input Operation Example 3) Input Operation Example 3 is an example in which, when user U1 taps a body feature area, an AR object for selecting input characters is displayed on the tapped body feature area. Figures 8A and 8B are explanatory diagrams illustrating Input Operation Example 3. The AR object displayed in this example is an AR object that draws a cross key, in which multiple types of characters that can be input using keys assigned to the body feature area are arranged in a cross shape.

[0078] In Figure 8A, when user U1 touches the proximal phalanges F3B1 of the middle finger LF3 of the left hand U1HL with the index finger RF2 of the right hand U1HR, the selection object OBJ31B is displayed at the position of the proximal phalanges F3B1 of the middle finger LF3 of the left hand U1HL. More specifically, with the selection object OBJ31B displayed, - Releasing the index finger RF2 of the right hand U1HR results in the input of 'g'. - Flicking the index finger RF2 of the right hand U1HR to the left results in the input of 'h'. - Flicking the index finger RF2 of the right hand U1HR upwards results in the input of 'i'. - Flicking the index finger RF2 of the right hand U1HR to the right results in the input of 'g' because there is no assigned character. - Flicking the index finger RF2 of the right hand U1HR downwards results in the input of '4'. Input is confirmed by either releasing the index finger RF2 of the right hand U1HR or by flicking it in either direction.

[0079] In the example in Figure 8B, the index finger RF2 of the right hand U1HR is flicked to the left (MV1), so 'h' is entered.

[0080] (Input Operation Example 4) Input Operation Example 4 is an example of setting uppercase or lowercase input mode using the movement (shape) of a physical characteristic part of user U1. Figure 9 is an explanatory diagram showing Input Operation Example 4.

[0081] When user U1 touches various parts of the left hand U1HL with the index finger RF2 of the right hand U1HR (for a predetermined period of time or longer), and the distal phalanx F1B3 of the left hand U1HL is bent at this time, a selection object OBJ31C is displayed at the touched location. Unlike the selection object OBJ31B, the selection object OBJ31C displayed when the distal phalanx F1B3 of the left hand U1HL is bent allows for direct selection of uppercase letters. In other words, it is possible to select uppercase or lowercase letters depending on the state of the distal phalanx F1B3 of the left hand U1HL.

[0082] In Figure 9, with the selection object OBJ31C displayed, the following inputs are possible: - Releasing the index finger RF2 of the right hand U1HR results in the input of 'G'. - Flicking the index finger RF2 of the right hand U1HR to the left results in the input of 'H'. - Flicking the index finger RF2 of the right hand U1HR upwards results in the input of 'I'. - Flicking the index finger RF2 of the right hand U1HR to the right results in the input of 'G' because there is no assigned character. - Flicking the index finger RF2 of the right hand U1HR downwards results in the input of '4'. - Releasing the index finger RF2 of the right hand U1HR or flicking it in either direction confirms the input.

[0083] In the example above, if the distal phalanx F1B3 of the left hand U1HL is extended, the selection object OBJ31B will be displayed at the touched position. When the selection object OBJ31B is displayed with the distal phalanx F1B3 extended, lowercase letters can be selected directly.

[0084] Furthermore, in Japanese mode as well, it may be possible to select uppercase or lowercase letters depending on the state of the distal phalanx F1B3 of the thumb of the left hand U1HL. As a modified example, it may be possible to select uppercase or lowercase letters depending on the state of the distal phalanx RF1 of the thumb of the right hand U1HR.

[0085] Furthermore, the user may choose between uppercase and lowercase letters depending on the state of the fingers on the left hand U1HL that are different from the area touched by the index finger RF2 of the right hand U1HR. That is, if the area touched by the index finger RF2 of the right hand U1HR is any of the proximal phalanx F3B1, middle phalanx F3B2, or distal phalanx F3B3 of the middle finger of the left hand U1HL, the user may choose between uppercase and lowercase letters depending on the state of the other finger (index finger LF2, ring finger LF4, or little finger LF5) (whether it is bent or straight).

[0086] Furthermore, the selection of uppercase or lowercase letters for input characters can be made by determining whether the distal phalanx F1B3 of the thumb of the left hand U1HL is bent when the user U1 touches various parts of the left hand U1HL (for a predetermined time or longer) with the index finger RF2 of the right hand U1HR, as described above. Alternatively, the selection of uppercase or lowercase letters for input characters can be made by determining whether a finger on the left hand U1HL that is different from the part touched by the index finger RF2 of the right hand U1HR is bent when the user U1 touches various parts of the left hand U1HL (for a predetermined time or longer). For example, the selection of uppercase or lowercase letters for input characters can be made by determining whether the little finger LF5 of the left hand U1HL is bent when the index finger RF2 of the right hand U1HR touches the middle phalanx F3B2 of the middle finger LF3 of the left hand U1HL.

[0087] Furthermore, when user U1 touches (or has touched) various parts of the left hand U1HL with the index finger RF2 of the right hand U1HR (for a predetermined time or longer), the user may select uppercase or lowercase letters of the input characters by touching from the front (with the index finger RF2 of the right hand U1HR moving from the front to the back in the depth direction of the diagram) or by touching the side of the finger (with the index finger RF2 of the right hand U1HR moving from the top to the bottom in the diagram, or moving within the paper of the drawing).

[0088] (Input Operation Example 5) Input operation example 5 is an example in which, when the first physical feature area of ​​user U1 is touched, a keypad with multiple types of characters that can be input using the first physical feature area arranged in a matrix is ​​displayed on the second physical feature area. It is preferable that the first and second physical feature areas are different areas, and that the first physical feature area has a larger area than the second physical feature area. Figures 10A and 10B are explanatory diagrams showing input operation example 5.

[0089] When user U1 touches any part of the left hand U1HL with the index finger RF2 of the right hand U1HR (for a predetermined period of time or longer), a selection object OBJ31D is displayed at the base of the four fingers F0B3 of the palm print area LF0.

[0090] In Figure 10A, when the index finger RF2 of the right hand U1HR touches the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL, a selection object OBJ31D corresponding to the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL is displayed at the base of the four fingers F0B3 of the palm print LF0. With the selection object OBJ31D displayed, tapping the position of the base of the four fingers F0B3 corresponding to the selection object OBJ31D with the index finger RF2 or thumb RF1 of the right hand U1HR inputs the character at the tapped position.

[0091] In Figure 10B, the letter 'h' is entered when the index finger RF2 of the right hand U1HR taps the position F0B3 of the base of the four fingers corresponding to the letter 'h' on the selection object OBJ31D.

[0092] Furthermore, if you tap or touch another part of the left hand U1HL with the index finger RF2 of the right hand U1HR, the processing corresponding to the touch on the proximal phalanx F3B1 of the middle finger LF3 of the left hand U1HL is canceled, and a selection object corresponding to the newly tapped or touched area is displayed.

[0093] Input is confirmed by a tap operation on the selection object OBJ31D using the index finger RF2 or thumb RF1 of U1HR.

[0094] (Input Operation Example 6) Input Operation Example 6 is an example of an input operation that implements a so-called backspace function, which deletes only the character immediately preceding the cursor position (corresponding to the character input position) using the movement of a physical characteristic part while the user U1 is not touching the screen. Figure 11 is an explanatory diagram showing Input Operation Example 6.

[0095] When user U1 moves the distal phalanx F1B3 of the thumb of the left hand U1HL from an extended state to a bent state MV2 while the index finger RF2 of the right hand U1HR is not tapping or touching any part of the left hand U1HL, the system may perform a backspace operation to delete the character immediately preceding the cursor position. In addition, to indicate that the delete operation has been accepted, an AR object OBJ13 may be displayed as shown in Figure 11.

[0096] Figure 12 is a flowchart showing the flow of the control method for the virtual input device executed in the head-mounted information display device 100 according to this embodiment.

[0097] The basic operation function unit 1101 checks the control status of various applications by the application control unit 1102 and confirms whether the main operation application is in text input mode. If the main operation application is in text input mode (S101: Yes), the process proceeds to S102. If the main operation application is not in text input mode (S101: No), the process ends.

[0098] The image acquisition unit 1111 activates the rear camera 133 (S102).

[0099] The image acquisition unit 1111 captures and acquires images of the user U1's hands (left hand U1HL and right hand U1HR) via the rear camera 133 (S103).

[0100] The input operation discrimination unit 1112 analyzes the captured video acquired in the processing of S103 and recognizes the left hand U1HL and the right hand U1HR. In this step, the input operation discrimination unit 1112 performs object recognition processing on the captured image and analyzes the recognition of the body feature area of ​​the user's left hand U1HL and the movement of the right hand U1HR (S104). In this step, the unit compares a first distance from the head-mounted information display device 100 measured by the depth sensor 164 to the body feature area of ​​the left hand U1HL with a second distance from the head-mounted information display device 100 to the fingertip of the right hand U1HR (corresponding to the pointing object). If the first distance and the second distance fall within a range where they can be considered the same, the unit determines that the fingertip of the right hand has touched or tapped the body feature area of ​​the left hand.

[0101] The input operation discrimination unit 1112 discriminates the input character based on the analysis results from the processing in S104 (see Figures 4 to 11 for details), inputs the discriminated character into the main operation application controlled by the application control unit 1102, and displays the discriminated character in the character input field displayed on the display 131 based on the control of the display control unit 1113 (S105).

[0102] The main operation application controlled by the application control unit 1102 checks whether to continue in character input mode. If the character input mode is to be continued (S106: No), the process returns to S103. If the character input mode is to be terminated (S106: Yes), the process proceeds to S107.

[0103] The image acquisition unit 1111 disables the rear camera 133 and terminates the process (S107).

[0104] According to this embodiment, keys for a virtual input device can be assigned to the user's physical characteristics, allowing key input by touching or tapping those physical characteristics. Since the user can feel the sensation of touching their own body parts, key touches can be confirmed without visual inspection, eliminating the instability of operation caused by the loss of tactile feedback when using a virtual input device.

[0105] Furthermore, even when a game involves viewing a virtual reality experience with movement and there are no real-world objects around the user that the user can touch, assigning keys to the user's own physical characteristics can provide a tactile sensation when operating a virtual input device.

[0106] Furthermore, since you can touch parts of your own body without necessarily looking, it makes touch typing easier.

[0107] Furthermore, in addition to assigning keys to the user's physical characteristics, it is possible to switch input modes using the movement of those physical characteristics, allowing for smooth switching of input modes while performing key touch operations.

[0108] Furthermore, by displaying AR objects (selection objects) that indicate the keys assigned to the body features overlaid on the body features, input operations can be performed while viewing the key layout. Alternatively, by displaying AR objects (confirmation objects) that indicate the entered keys, it is possible to confirm whether the input was correct.

[0109] <Modification> As shown above, each part of the left hand was given as an example of a function related to text input, but the functions that can be assigned to physical feature parts are not limited to text input functions. For example, with the left palm facing the face and the index finger, middle finger, ring finger, and little finger held roughly horizontally, the proximal, middle, and distal phalanges of the index finger, the proximal, middle, and distal phalanges of the middle finger, the proximal, middle, and distal phalanges of the ring finger, and the proximal, middle, and distal phalanges of the little finger may be considered as a keypad with 3 rows horizontally and 4 columns vertically, and a predetermined function may be assigned to them.

[0110] Alternatively, with the ring and little fingers folded (clenched), the proximal, middle, and distal phalanges of the index finger and the proximal, middle, and distal phalanges of the middle finger may be treated as a 3x2 keypad and assigned predetermined functions to them.

[0111] In addition, it is possible to use only one (or three) fingers to simulate a keypad with 3 rows horizontally and 1 (or 3) vertically.

[0112] Furthermore, due to individual differences, if it is difficult to hold the index, middle, ring, and little fingers in a roughly horizontal position with the left palm facing the face, the index, middle, ring, and little fingers may be held in a roughly vertical position with the left palm facing the face. In this case, the proximal, middle, and distal phalanges of the index finger, the proximal, middle, and distal phalanges of the middle finger, the proximal, middle, and distal phalanges of the ring finger, and the proximal, middle, and distal phalanges of the little finger can be considered as a 4x3 keypad and assigned predetermined functions.

[0113] Furthermore, the specific body parts to which keys are assigned may be changed as appropriate according to user convenience. For example, the back of the hand may be used instead of the palm. As an example, even when the back of the left hand is facing the face and the index, middle, ring, and little fingers are held roughly horizontally, it is possible to treat the proximal, middle, and distal phalanges of the index finger, the proximal, middle, and distal phalanges of the middle finger, the proximal, middle, and distal phalanges of the ring finger, and the proximal, middle, and distal phalanges of the little finger as a 3x4 keypad and assign predetermined functions to them. However, in this case, the proximal, middle, and distal phalanges of the little finger would be the top row, and the proximal, middle, and distal phalanges of the index finger would be the bottom row.

[0114] Furthermore, the functions of the left and right hands may be reversed.

[0115] Furthermore, in the example of assigning keys to the left and right forearms, for example, physical features that can be detected from the camera's captured image, such as vein patterns, moles, or areas where the skeleton is prominent, may be used. Also, the term "physical features" is not limited to the user's body surface, but is intended to include the outer surface of the clothing the user is wearing. For example, in the example of assigning keys to the forearms, if the forearms are covered with clothing, the pattern of the clothing may be used as the target for key assignment. In this case, before S101 in Figure 12, the key assignment process should be executed as S100, where the pattern of the user's clothing captured by the rear camera 133 is touched, and a process is executed to register the function to be assigned to the touched area. This process is performed by the input operation discrimination unit 1112, and "key assignment data" associating the touched area with the function assigned to it is saved in the temporary storage area 1199. Then, in the processes from S101 onward, the input operation discrimination unit 1112 refers to the "key assignment data" and implements the function corresponding to the touched area. Furthermore, the same applies when the user is wearing gloves. Keys may be assigned to body features such as the pattern on the glove, wrinkles or scratches that have formed from prolonged use of the glove, or key assignments may be made to parts of the glove that correspond to different parts of the user's palm inside the glove, similar to Figures 5A and 5B.

[0116] Furthermore, in the example of assigning keys to parts of the user's left hand as described below, if, for example, the user has fractured their left thumb and the head-mounted information display device 100 cannot detect the characteristic part of the left thumb, the key assigned to the left thumb in S100 may be temporarily assigned to the left forearm, wrist, etc.

[0117] Furthermore, although the above description was based on a Japanese keyboard layout, the virtual input device according to this embodiment is not limited to a Japanese keyboard layout, but can also be applied to other languages, such as a Korean keyboard layout. Figures 13A, 13B, and 13C show examples of Korean keyboard layouts.

[0118] As shown in Figures 13A, 13B, and 13C, in the case of a Korean keypad with three horizontal rows and four vertical columns that implements a mechanism for combining Korean consonants and vowels to form a single character, this embodiment can also be applied to a virtual Korean input device by assigning each of the bald parts of the body of the index finger, middle finger, ring finger, and little finger to the respective Korean keys, even when the left palm or back of the hand is facing the face and the four fingers (index, middle, ring, and little fingers) are held roughly horizontally.

[0119] Alternatively, a handwriting pad function may be assigned to the base of the four fingers, F0B3. Figures 14A and 14B show examples of operations using the handwriting pad function.

[0120] In an example of operation using the handwriting pad function, the handwriting mode is started when user U1 taps the base of the four fingers F0B3 of the left hand U1HL with the index finger RF2 of the right hand U1HR, while none of the index finger LF2, middle finger LF3, ring finger LF4, or little finger LF5 of the left hand U1HL are touching anything. In handwriting mode, as shown in Figure 14A, an AR object OBJ11A is displayed to indicate that the current input mode is handwriting mode, and the handwriting pad OBJ31E is displayed on the base of the four fingers F0B3. The handwriting pad OBJ31E consists of a pad section for handwriting input and a function button section, and the function button section displays, for example, a back button and an erase button.

[0121] User U1 writes characters by moving the index finger RF2 of the right hand U1HR on the four-finger base F0B3 corresponding to the pad portion of the handwriting pad OBJ31E, while keeping it in contact with the four-finger base F0B3 of the left hand U1HL. As shown in Figure 14B, the trajectory of the index finger RF2 of the right hand U1HR (a rightward arrow in the example shown) is displayed on the pad portion of the handwriting pad OBJ31E, and candidate characters OBJ31F, inferred from the trajectory of the index finger RF2 of the right hand U1HR, are displayed in the area from the thenar eminence F0B1 to the hypothenar eminence F0B5. User U1 can confirm character input by tapping one of the candidate characters OBJ31F with the index finger RF2 of the right hand U1HR to select it.

[0122] Thus, in addition to assigning specific keys to characteristic parts of the fingers for character input, it is also possible to perform input operations efficiently by using the palm area as a free input area.

[0123] Although embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, the embodiments described above were explained in detail for the purpose of making the present invention easy to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. In addition, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0124] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is executed may also be changed.

[0125] The functions of the present invention described above may be implemented in hardware, in whole or in part, by designing them, for example, using integrated circuits, general-purpose processors, or application-specific processors. A processor includes transistors and other circuits and is considered a circuit or processing circuit. Alternatively, the functions may be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program that realizes each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used in combination. In addition, some or all of each function may be implemented by a server. The server only needs to be able to perform functions in cooperation with other components via communication, and its form is not limited to, for example, a local server, cloud server, edge server, or network service. Information such as programs, tables, and files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored in a device on a communication network.

[0126] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0127] The above embodiment includes the following invention: (Note 1) An information processing device comprising: a processor; an out-camera; and a depth sensor, wherein the processor analyzes an image generated by the out-camera imaging the area around the information processing device to identify a selected body feature area among the body feature areas of the user of the information processing device; compares a first distance to the body feature area detected by the depth sensor with a second distance to an indicator that touches or taps the body feature area to determine whether the indicator has touched or tapped the body feature area; and, if it is determined that the indicator has touched or tapped the body feature area, refers to pre-created body key assignment data showing the correspondence between the body feature area and the key of a virtual input device assigned to that body feature area, and accepts an input operation of the key of the virtual input device assigned to the body feature area that the indicator has touched or tapped.

[0128] (Note 2) An information processing device comprising: a processor and an out-camera, wherein the processor analyzes an image generated by the out-camera imaging the area surrounding the information processing device to identify the movement or shape of a body feature part of the user of the information processing device, refers to pre-created body key assignment data showing the correspondence between the movement or shape of the body feature part and the key of a virtual input device assigned to the movement or shape of the body feature part, and accepts input operations of the key of the virtual input device assigned to the movement or shape of the body feature part.

[0129] (Note 3) A method for controlling a virtual input device, comprising: a step of a processor analyzing an image generated by an out-camera imaging the area around the information processing device to identify a selected body feature area among the body feature areas of the user of the information processing device; a step of comparing a first distance detected by a depth sensor from the information processing device to the body feature area and a second distance from the information processing device to an indicator that touches or taps the body feature area to determine whether the indicator has touched or tapped the body feature area; and, if it is determined that the indicator has touched or tapped the body feature area, a step of referring to pre-created body key assignment data showing the correspondence between the body feature area and the key of the virtual input device assigned to that body feature area, and accepting an input operation of the key of the virtual input device assigned to the body feature area that the indicator has touched or tapped.

[0130] (Note 4) A method for controlling a virtual input device, comprising: a step of a processor analyzing an image generated by an out-camera capturing images of the area surrounding the information processing device to identify the movement or shape of a user's physical characteristic part of the information processing device; and a step of referring to pre-created part key assignment data that shows the correspondence between the movement or shape of the physical characteristic part and the key of the virtual input device assigned to the movement or shape of the physical characteristic part, and accepting an input operation of the key of the virtual input device assigned to the movement or shape of the physical characteristic part.

[0131] 100: Head-mounted information display device, 101: Processor, 102: Bus, 103: RAM, 110: Storage, 120: User operation interface, 121: Operation keys, 122: Touch sensor, 122L: Left touch sensor, 130: Image processing unit, 131: Display, 131L: Left display, 131R: Right display, 132: Image signal processing unit, 133: Outer camera, 133L: Left outer camera 133R: Right rear camera, 134: Front camera, 140: Audio processing unit, 141: Audio output unit, 141L: Left stereo speaker, 142: Audio signal processing unit, 143: Microphone, 150: Positioning sensor, 160: Sensor group, 161: Gyro sensor, 162: Geomagnetic sensor, 163: Accelerometer, 164: Depth sensor, 164L: Left depth sensor, 164R: Right depth sensor, 170: Communication I / F, 171: LAN I / F, 172: BLUETOOTH I / F, 180: Extended I / F, 1001: Basic operation program, 1002: Application program, 1009: Data storage area, 1101: Basic operation function unit, 1102: Application control unit, 1111: Image acquisition unit, 1112: Input operation discrimination unit, 1113: Display control unit, 1199: Temporary storage area, F0B1: Thenar eminence, F0B3: Base of the four fingers, F0B5: Hypothenar eminence, F1B1: Proximal phalanx of the thumb, F1B3: Distal phalanx of the thumb, F1J1: Proximal phalanx of the thumb, F1J3: Distal phalanx of the thumb, F2B1: Proximal phalanx of the index finger, F2B2: Middle phalanx of the index finger, F2B3: Distal phalanx of the index finger, F2J1: Base phalanx of index finger, F2J2: Middle phalanx of index finger, F2J3: Distal index phalanx, F3B1: Base phalanx of middle finger, F3B2: Middle phalanx of middle finger, F3B3: Distal middle phalanx, F3J1: Base phalanx of middle finger, F3J2: Middle phalanx of middle finger, F3J3: Distal middle phalanx. F4B1: ring finger proximal phalanx, F4B2: ring finger middle phalanx, F4B3: ring finger distal phalanx, F4J1: ring finger base phalanx, F4J2: ring finger middle phalanx, F4J3: ring finger distal phalanx, F5B1: little finger base phalanx, F5B2: little finger middle phalanx, F5B3: ring finger distal phalanx, F5J1: Base phalanx of little finger, F5J2: Middle phalanx of little finger, F5J3: End phalanx of little finger, LF0: left hand palm print, LF1: left hand thumb, LF2: left hand index finger, LF3: left hand middle finger, LF4: left hand ring finger, LF5: left hand little finger, RF1: right hand thumb, RF2: right hand index finger, U1: user, U1HL: left hand, U1HR: right hand MV1: flick operation, MV2: bending motion,OBJ01: AR object, OBJ05: AR object, OBJ11: AR object, OBJ11A: AR object, OBJ13: AR object, OBJ21: AR object, OBJ22: AR object, OBJ23: AR object, OBJ31: AR object, OBJ31A: Confirmation object, OBJ31B: Selection object, OBJ31C: Selection object, OBJ31D: Selection object, OBJ31E: Handwriting pad object, OBJ31F: Input candidate character object, OBJ32: AR object, OBJ33: AR object, OBJ41: AR object, OBJ42: AR object, OBJ43: AR object, OBJ51: AR object, OBJ52: AR object, OBJ53: AR object

Claims

1. An information processing device comprising: a processor; an out-camera; and a depth sensor, wherein the processor analyzes an image generated by the out-camera imaging the area surrounding the information processing device to identify a selected body feature area among the body feature areas of the user of the information processing device; compares a first distance to the body feature area detected by the depth sensor with a second distance to an indicator that touches or taps the body feature area to determine whether the indicator has touched or tapped the body feature area; and, if it is determined that the indicator has touched or tapped the body feature area, refers to pre-created body key assignment data showing the correspondence between the body feature area and the key of a virtual input device assigned to that body feature area, and accepts an input operation of the key of the virtual input device assigned to the body feature area that the indicator has touched or tapped.

2. An information processing device according to claim 1, wherein the physical characteristic part is a part of the user's hand.

3. An information processing device according to claim 2, wherein the physical characteristic part is a part of the user's left hand and the indicator is a finger of the user's right hand, or the physical characteristic part is a part of the user's right hand and the indicator is a finger of the user's left hand.

4. An information processing device according to claim 2, wherein in the part key assignment data, one key of the virtual input device is assigned to at least one or more joints of the user's left or right fingers.

5. An information processing device according to claim 2, wherein, in the part key assignment data, one key of the virtual input device is assigned to at least one of the thenar eminence or hypothenar eminence of the user's palm.

6. An information processing device according to claim 2, wherein the key has a function for inputting multiple types of characters, the part key assignment data defines a correspondence between the number of taps of the physical feature part and the input character corresponding to the number of taps, and the processor detects the number of consecutive taps within a predetermined time after the part of the user's hand used as the physical feature part is first touched or tapped, based on the captured image, the first distance, and the second distance, and accepts an input operation for a character corresponding to the detected number of taps by referring to the part key assignment data.

7. An information processing device according to claim 2, further comprising a display, wherein the keys have a function for inputting multiple types of characters, the part key assignment data defines a correspondence between the number of taps of the body feature part and the input character corresponding to that number of taps, and the processor detects the number of times the user's hand part has been tapped consecutively within a predetermined time based on the captured image, the first distance, and the second distance, and accepts input according to that number.

8. An information processing apparatus according to claim 7, wherein the processor displays an augmented reality object representing the key superimposed on the body feature area to which the key of the virtual input device is assigned.

9. An information processing device according to claim 7, wherein the processor displays an augmented reality object representing the key superimposed on a part of the user's left hand or right hand that is different from the body feature part to which the key of the virtual input device is assigned.

10. An information processing device according to claim 2, further comprising a display, wherein the keys have a function for inputting a plurality of types of characters, and the processor, upon detecting that a part of the user's hand has been touched or tapped based on the captured image, the first distance, and the second distance, displays an augmented reality object superimposed on the user's hand, which is visible through the display, which has a cross key drawn on it, in which a plurality of types of characters that can be input using the keys of the virtual input device assigned to the touched or tapped body feature part have been arranged in a cross shape, and upon detecting, within a predetermined time after the augmented reality object has been displayed, a flick operation or a predetermined number of tap operations performed on the surface of the user's hand on which the augmented reality object is superimposed, based on the captured image, the first distance, and the second distance, accepts an input operation for the character selected by the detected flick operation or tap operation.

11. An information processing apparatus according to claim 1, further comprising a display, wherein the physical feature part is a part of the user's hand, and the processor continuously displays an augmented reality object, which is an input mode switching key for setting the switching of character input, superimposed on the user's hand as seen through the display, while accepting input operations of the virtual input device.

12. An information processing device according to claim 2, further comprising a display, wherein the keys have a function for inputting a plurality of types of characters, and the processor, upon detecting that a part of the user's hand has been touched or tapped based on the captured image, the first distance, and the second distance, displays an augmented reality object superimposed on the user's hand, which is visible through the display, which has an augmented reality object on which a keypad is drawn, in which a plurality of types of characters that can be input using the keys of the virtual input device assigned to the touched or tapped body feature part are arranged in a matrix. The information processing device, upon detecting that the part of the user's hand on which the characters displayed in the augmented reality object are superimposed has been tapped, accepts an input operation for the tapped character.

13. An information processing device comprising: a processor and an out-camera, wherein the processor analyzes an image generated by the out-camera imaging the area surrounding the information processing device to identify the movement or shape of a body feature part of the user of the information processing device, refers to pre-created body key assignment data showing the correspondence between the movement or shape of the body feature part and the key of a virtual input device assigned to the movement or shape of the body feature part, and accepts input operations of the key of the virtual input device assigned to the movement or shape of the body feature part.

14. An information processing device according to claim 13, wherein in the part key assignment data, an uppercase input mode and a lowercase input mode are assigned to the extended state and the bent state of the user's hand, which is used as the physical characteristic part, respectively, in English alphabet mode, and the processor analyzes the captured image to identify the bent and extended state of the user's hand, and accepts input operations of the keys of the virtual input device in the uppercase input mode or lowercase input mode corresponding to the bent and extended state of the user's hand.

15. An information processing device according to claim 13, wherein in the part key assignment data, the action of bending the part of the user's hand used as the physical feature part from an extended state is assigned to a backspace function that deletes the character immediately preceding the cursor position indicating the character input position, and the processor, upon detecting the action by analyzing the captured image, executes the backspace function.

16. A method for controlling a virtual input device, comprising: a step of a processor analyzing an image generated by an out-camera capturing images of the area surrounding an information processing device to identify a selected body feature area among the body feature areas of the user of the information processing device; a step of comparing a first distance detected by a depth sensor from the information processing device to the body feature area and a second distance from the information processing device to an indicator that touches or taps the body feature area to determine whether the indicator has touched or tapped the body feature area; and, if it is determined that the indicator has touched or tapped the body feature area, referring to pre-created body key assignment data showing the correspondence between the body feature area and the key of the virtual input device assigned to that body feature area, and accepting an input operation of the key of the virtual input device assigned to the body feature area touched or tapped by the indicator.

17. A method for controlling a virtual input device, comprising: a step of a processor analyzing captured video generated by an out-camera capturing images of the area surrounding the information processing device to identify the movement or shape of a user's physical characteristic part of the information processing device; and a step of referring to pre-created part key assignment data that shows the correspondence between the movement or shape of the physical characteristic part and the key of the virtual input device assigned to the movement or shape of the physical characteristic part, and accepting an input operation of the key of the virtual input device assigned to the movement or shape of the physical characteristic part.

Images