Controller, information processing system, and information processing method

Abstract

This information processing system has a controller and an image control unit. The controller has a touch operation surface on which a touch operation can be performed. The image control unit displays a virtual input surface in the vicinity of a 3D object to be touch-operated. The image control unit changes the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch operation surface. The image control unit fixes the relative positions and orientations of the virtual input surface and the 3D object on the basis of a capture command from the controller. The image control unit changes the position and orientation of the 3D object according to the movement of an arm holding the controller.

Description

Controller, Information Processing System, and Information Processing Method

[0001] The present invention relates to a controller, an information processing system, and an information processing method.

[0002] Editing 3D data using a pen and a tablet, which are 2D operations, is difficult in terms of the recognition and instruction of "how to assign 2D operations on the tablet to operations on a 3D object". On the other hand, 3D operations in the air are intuitive both in terms of recognition and instruction because they are operations in the same dimension as the target 3D object. However, since there is no means to support the hand during operation and the operation is in a floating state in the air, the operation tends to be unstable and the degree of fatigue is high.

[0003] Japanese Patent Application Laid-Open No. 2023-164499

[0004] Conventional VR controllers focus on versatility and immersion experience. Therefore, they do not sufficiently address physical support for creation purposes such as drawing (for example, device configurations aimed at "high precision" and "fatigue reduction"). The cause of fatigue in air operations is being forced into an unstable working posture over a long period of time. In order to perform stable input, it is necessary to realize a natural working posture similar to the real space on the UI.

[0005] Therefore, the present disclosure proposes a controller, an information processing system, and an information processing method capable of enhancing the stability of input operations in air operations and reducing the degree of fatigue.

[0006] According to the present disclosure, there is provided a controller including: a board unit having a touch operation surface capable of touch operations; and an operation detection unit that detects a capture operation for capturing, in the vicinity of a virtual input surface whose position and orientation change in conjunction with changes in the position and orientation of the touch operation surface, a 3D object that is the target of the touch operation, in a state where the relative position and orientation between the 3D object and the virtual input surface are fixed.

[0007] According to this disclosure, an information processing system is provided, comprising: a controller having a touch-operable touch surface; and a video control unit that displays a virtual input surface near a 3D object to be touched, changes the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch surface, fixes the relative position and orientation of the virtual input surface and the 3D object based on a capture command from the controller, and changes the position and orientation of the 3D object in accordance with the movement of the arm holding the controller. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing system is performed by a computer.

[0008] This figure shows an example of a conventional 3D creation work scene. This figure shows an example of a 3D creation work scene according to the present disclosure. This figure shows an example of the configuration of an information processing system. This figure shows an example of specifying a pointing range using a grip board. This figure shows an example of the relationship between a virtual input surface and a pointing range. This figure shows an example of the relationship between a virtual input surface and a pointing range. This figure shows an example of treating the pointing range as a virtual input surface and performing a direct touch operation on the pointing range. This figure shows an example of stopping the detection and display of the pointing range. This figure illustrates an auxiliary image of the pointing range. This figure shows an example of specifying a pointing range for a remote 3D object. This figure shows how a touch operation is performed on a remote 3D object. This figure shows an example of a grip board integrated with a pick mouse. This figure shows a five-view drawing of the grip board. This figure shows the grip board with the board part and grip part separated. This figure shows how a touch operation is performed using a pick mouse. This figure shows an example of adjusting the angle of the grip board. This figure shows an example where there are multiple touch operation surfaces. This figure shows a touch operation surface having a pin array structure. This figure shows a modified method of holding the board part. This figure shows a modified method of holding the board part. This figure shows a modified version of an information processing system. This figure shows an example of the hardware configuration of an information processing system.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will proceed in the following order: [1. 3D Creation] [2. Configuration of the Information Processing System] [3. Specifying the Pointing Range] [4. Auxiliary Images for the Pointing Range] [5. Specifying the Pointing Range for Remote 3D Objects] [6. Modified Grip Boards] [6-1. Grip Board Integrated with Pick Mouse] [6-2. Example of Grip Board Angle Adjustment] [6-3. Example with Multiple Touch Operation Surfaces] [6-4. Touch Operation Surface with Pin Array Structure] [6-5. Others] [7. Modified Information Processing System] [8. Hardware Configuration Example]

[0011] [1. 3D Creation] Figure 1 shows an example of a conventional 3D creation workflow.

[0012] 3D creation refers to the process of performing input and processing on virtual objects (3D objects OB) within a virtual space (VS). The left side of Figure 1 shows a typical 3D creation workflow in a PC environment. The right side of Figure 1 shows a typical 3D creation workflow in an XR (Extended Reality) environment.

[0013] In the example on the left of Figure 1, a 3D object OB is displayed as a 2D image on the PC monitor. The user US uses various controllers such as a mouse MS, a tablet terminal TL, a stylus pen 40, and a left-hand device LD to perform operations such as pointing, input, and resizing on the 3D object OB to be processed.

[0014] In the example on the right side of Figure 1, the 3D object OB is displayed as a 3D image by the HMD (Head Mounted Display) 20. The user US uses various controllers such as the VR controller CN and mouse MS to perform operations such as pointing, input, and resizing on the 3D object OB to be processed.

[0015] In a PC environment, pointing and input tasks are performed using desktop operations. Desktop operations refer to pointing operations performed by moving a controller on a support surface such as a desk. In desktop operations, the degree of freedom of operation is limited to two dimensions (two axes), but because the hands and arms operating the controller are supported by the support surface, fatigue is reduced.

[0016] In an XR environment, pointing and input operations are performed through aerial manipulation. Aerial manipulation refers to pointing operations performed by holding the controller in one's hand without support from a support surface. While aerial manipulation offers three-dimensional (three-axis) freedom of movement, there is no means of supporting the hands or arms during operation, resulting in operation while floating in the air. Because the work must be performed in accordance with the position and orientation of the 3D object (OB), the operation tends to be unstable and fatigue-inducing.

[0017] This disclosure was made in view of the above-mentioned problems. This disclosure proposes a method that can improve the stability of operations in aerial manipulation and reduce fatigue. Figure 2 shows an example of a 3D creation work scene according to this disclosure.

[0018] In this disclosure, a controller called a grip board 50 is used to specify a pointing range on a 3D object OB. The pointing range refers to the area on the 3D object OB that is the target of touch operation. For example, the user US brings the grip board 50 close to the part of the 3D object OB to be processed and performs a capture operation. As a result, the part of the 3D object OB facing the grip board 50 is acquired as the pointing range.

[0019] The capture operation fixes the pointing range of the 3D object OB to the touch operation surface TC of the grip board 50. The user US performs input and other processing on the pointing range by touching the touch operation surface TC using a stylus 40 or the like. The pointing range does not change even if the position and orientation of the grip board 50 change. The user US can perform touch operations while holding the grip board 50 in a free position. The user US can perform input via the touch operation surface while the 3D object OB has been captured and held in an easy-to-handle position. Since there is no need to work in accordance with the orientation of the 3D object OB, stable input is possible and fatigue is reduced.

[0020] Touch operation can be performed using the same principles as a normal touch panel, such as capacitive, electromagnetic, or thin-film contact methods. The positional relationship between the 3D object OB and the touch operation surface TC can be detected by analyzing the image obtained by the HMD20's built-in camera. For example, LED markers are installed on the edge of the grip board 50, and the position and orientation of the touch operation surface TC can be detected by analyzing the position of the LED markers. Alternatively, the grip board 50 can be equipped with a SLAM (Simultaneous Localization and Mapping) camera capable of spatial position recognition on its own to acquire the positional relationship.

[0021] [2. Configuration of the Information Processing System] The information processing described herein is performed by an information processing system 100 as shown in Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing system 100.

[0022] For example, the information processing system 100 includes a stylus 40, a grip board 50, and an HMD 20. The HMD 20 displays a 3D object OB in a virtual space VS. The user US uses the grip board 50 to point to the 3D object OB. The user US uses the stylus 40 and other devices to perform processing such as inputting data to or processing the 3D object OB.

[0023] The stylus pen 40 is a pen-shaped controller. The stylus pen 40 includes a pen input detection unit 41, a 6DoF position detection unit 42, an operation detection unit 43, a communication control unit 44, an operation control unit 45, and a power supply control unit 46. The pen input detection unit 41 detects the pen pressure during touch operation. Pen pressure detection is performed using a pressure sensor or the like built into the tip of the stylus pen 40.

[0024] The 6DoF position detection unit 42 detects the 6DoF position of the stylus 40. The 6DoF position is detected using information from an inertial sensor or other sensor built into the stylus 40. The operation detection unit 43 detects user operations on the operation components (operation buttons, sticks, trackballs, wheels, etc.) of the stylus 40.

[0025] The motion control unit 45 acquires touch pen detection information such as the pressure of the touch pen 40, the 6DoF position of the touch pen 40, and operation information for the touch pen 40's operating member. The motion control unit 45 controls the operation of the touch pen 40 based on the touch pen detection information. The communication control unit 44 outputs the touch pen detection information to the HMD 20. The power supply control unit 46 controls the power supply of the touch pen 40.

[0026] The grip board 50 is a board-type controller having a touch-operable touch surface TC. The grip board 50 includes a pen input detection unit 51, a 6DoF position detection unit 52, an operation detection unit 53, a communication control unit 54, an operation control unit 55, and a power supply control unit 56. The pen input detection unit 51 detects the 2D position of the touch pen 40 on the grip board 50 during touch operation. The 2D position is detected based on changes in capacitance between the touch pen 40 and the grip board 50.

[0027] The 6DoF position detection unit 52 detects the 6DoF position of the grip board 50. The 6DoF position is detected using information from an inertial sensor or other device built into the grip board 50. The operation detection unit 53 detects user operations on the operating components (operation buttons, sticks, trackballs, wheels, etc.) of the grip board 50.

[0028] The motion control unit 55 acquires the 2D position of the grip board 50, the 6DoF position of the grip board 50, and operation information for the operating member of the grip board 50 as grip board detection information. The motion control unit 55 controls the operation of the grip board 50 based on the grip board detection information. The communication control unit 54 outputs the grip board detection information to the HMD 20. The power supply control unit 56 controls the power supply of the grip board 50.

[0029] The HMD 20 includes a communication control unit 21, an operation control unit 22, a power supply control unit 23, an image control unit 24, an audio control unit 25, an operation detection unit 26, and a controller position detection unit 27. The communication control unit 21 acquires touch pen detection information and grip board detection information from the touch pen 40 and the grip board 50.

[0030] The controller position detection unit 27 detects the position information of the HMD 20 as the controller position. The controller position is detected using information from the camera and inertial sensors built into the HMD 20. Technologies such as SLAM can be used for the calculation method. The operation detection unit 26 detects the user's (US) operation on the operating member built into the HMD 20.

[0031] The video control unit 24 acquires the position information of the HMD 20 and the user's (US) operation information on the HMD 20 as HMD detection information. Based on the HMD detection information, the video control unit 24 generates an image of the 3D object OB. Based on the HMD detection information, touch pen detection information, and grip board detection information, the video control unit 24 generates an image of the UI (User Interface) for pointing, moving, deforming, and processing the 3D object OB. The video control unit 24 displays the generated image on a display screen using a light source such as an SLD (Superluminescent diode).

[0032] The audio control unit 25 outputs audio in accordance with the video. The motion control unit 22 controls the operation of the HMD 20 based on HMD detection information. The power supply control unit 23 controls the power supply of the HMD 20.

[0033] [3. Specifying the Pointing Range] Figure 4 shows an example of specifying the pointing range using the grip board 50.

[0034] The video control unit 24 displays a 3D object OB in the virtual space VS. The movement of the grip board 50 can be recognized by the movement of the virtual board 50V. The virtual board 50V is a 3D object in the virtual space VS that mimics the grip board 50. The virtual board 50V exhibits the same movement as the grip board 50. The video control unit 24 displays the virtual board 50V at the position where the grip board 50 is located.

[0035] For example, user US moves the grip board 50 with their left hand to bring the virtual board 50V closer to the processing target area of ​​the 3D object OB. Once the virtual board 50V is in a position opposite the processing target area, user UV operates the operation buttons on the grip board 50 to confirm the position of the virtual board 50V. Once the position of the virtual board 50V is confirmed, the video control unit 24 acquires the area opposite the virtual board 50V as the pointing range RG.

[0036] For example, the video control unit 24 displays a virtual input surface VI on the virtual board 50V. The virtual input surface VI corresponds to the touch operation surface TC. The positional relationship between the virtual board 50V and the virtual input surface VI matches the positional relationship between the grip board 50 and the touch operation surface TV. The video control unit 24 matches the orientation of the virtual board 50V and the virtual input surface VI to the orientation of the grip board 50 and the touch operation surface TC. With this configuration, the operating posture in real space and the operating posture in the virtual space VS match. Therefore, confusion in perception and instructions is less likely to occur during operation in the virtual space VS.

[0037] The video control unit 24 displays a virtual board 50V and a virtual input surface VI near the 3D object OB that is the target of touch operation. The video control unit 24 changes the position and orientation of the virtual board 50V (virtual input surface VI) in conjunction with changes in the position and orientation of the grip board 50 (touch operation surface TC). The video control unit 24 displays the area on the 3D object OB obtained by projecting the virtual input surface VI onto the 3D object OB as the pointing range RG. The user US can position the virtual input surface VI relative to the 3D object OB while checking the pointing range RG.

[0038] The operation detection unit 53 detects an operation to determine the position of the virtual board 50V as a 3D object OB capture operation. A 3D object OB capture operation means an operation to capture the 3D object OB in the vicinity of the virtual input surface VI. The capture of the 3D object OB is performed while the relative position and orientation of the 3D object OB and the virtual input surface VI are fixed.

[0039] When a capture operation is detected, the grip board 50 outputs a capture command to the HMD 20. The video control unit 24 fixes the relative position and orientation of the virtual input surface VI and the 3D object OB based on the capture command from the grip board 50. The video control unit 24 changes the position and orientation of the 3D object OB in accordance with the movement of the arm holding the grip board 50.

[0040] The relative position and orientation of the virtual input surface VI and the 3D object OB are fixed. Therefore, the pointing range does not change even if the position and orientation of the grip board 50 are changed. The user US can perform touch operations in a relaxed posture while bringing the pointing range closer to their hand. The user US holds the grip board 50 with their left hand and operates the stylus 40 with their right hand. The touch operation is performed with the right and left hands supporting each other. As a result, touch operation is stable and precise work becomes easier.

[0041] Figures 5 and 6 show an example of the relationship between the virtual input plane VI and the pointing range RG.

[0042] The pointing range RG means the area on the surface of the 3D object OB when the virtual input surface VI is projected onto the 3D object OB. As the projection method, there are parallel projection, wrapping projection, etc. Fig. 5 shows an example of parallel projection. Fig. 6 shows an example of wrapping projection.

[0043] Parallel projection means projecting each point on the outer contour line of the virtual input surface VI parallel to the normal line of the virtual input surface VI onto the 3D object OB. The area surrounded by the point group projected onto the 3D object OB becomes the pointing range RG. Wrapping projection means attaching the virtual input surface VI along the surface shape of the 3D object OB. The area surrounded by the outer contour line of the virtual input surface VI attached to the 3D object OB becomes the pointing range RG.

[0044] The video control unit 24 displays the virtual pen 40V near the virtual input surface VI. The virtual pen 40V corresponds to the touch pen 40. The positional relationship between the virtual pen 40V and the virtual input surface VI is the same as the positional relationship between the touch pen 40 and the touch operation surface TC. The video control unit 24 makes the position and orientation of the virtual pen 40V coincide with the position and orientation of the touch pen 40.

[0045] Since the user US's vision is blocked by the HMD20, the user US cannot directly see the touch pen 40 and the clipboard 50. What the user US directly visually recognizes are the virtual pen 40V and the virtual board 50V displayed in the virtual space VS. Since the movements of the virtual pen 40V and the virtual board 50V coincide with the movements of the touch pen 40 and the clipboard 50 that the user US operates, the user US can perform touch operations without a sense of discomfort.

[0046] The video control unit 24 can stop displaying the virtual board 50V and the virtual input surface VI in conjunction with the acquisition of the capture command. This is because if the virtual board 50V and the virtual input surface VI exist in front of the pointing range RG, it may be annoying during touch operations. By stopping the display of the unnecessary virtual input surface VI, the processing load is reduced. When the display of the virtual input surface VI is stopped, the user US treats the pointing range RG as the virtual display surface VI.

[0047] FIG. 7 is a diagram showing an example in which the pointing range RG is treated as a virtual input surface VI and a touch operation is directly performed on the pointing range RG. The video control unit 24 displays a virtual pen 40V in the vicinity of the pointing range RG. The positional relationship between the virtual pen 40V and the pointing range RG coincides with the positional relationship between the touch pen 40 and the touch operation surface TC.

[0048] The video control unit 24 can stop detecting and displaying the pointing range RG until the virtual input surface VI approaches the 3D object OB. When the virtual input surface VI is in a state of being separated from the 3D object OB, it is considered that the possibility of the user US performing a capture operation is low. By omitting the process of detecting and displaying the pointing range RG in a situation where the capture operation is difficult to perform, the processing load is reduced.

[0049] Whether or not it is recognized that the virtual input surface VI has approached the 3D object OB is determined based on proximity conditions preset by the system developer. For example, the proximity condition can be set such that the virtual input surface VI has approached a position where the distance from the 3D object OB is equal to or less than a threshold value.

[0050] FIG. 8 is a diagram showing an example in which the detection and display of the pointing range RG are stopped. In the example of FIG. 8, the virtual board 50V is in a state of being recessed inside the 3D object OB. The video control unit 24 can display the virtual board 50V at an arbitrary position in the virtual space VS according to the movement of the hand holding the grip board 50. It is also allowed that the virtual board 50V is arranged at a position overlapping the 3D object OB. However, since the state where the virtual input surface VI is recessed inside the 3D object OB cannot be said to be "proximity", the detection and display of the pointing range RG are not performed.

[0051] [4. Auxiliary Image of Pointing Range] FIG. 9 is a diagram for explaining an auxiliary image AU of the pointing range RG.

[0052] The video control unit 24 displays an auxiliary image AU in the virtual space VS in parallel with the 3D display of the 3D object OB, allowing the user US to check their operation status in real time. The auxiliary image AU is displayed as a 2D image of the 3D object OB showing the real-time operation status of the pointing range RG by touch operation. The user US can perform touch operations while checking the operation status of the 3D object OB in real time.

[0053] In the example shown in Figure 9, the auxiliary image AU is displayed as an image of the 3D object OB viewed from the normal direction of the virtual input surface VI. The display screen 20A of the HMD 20 shows both the 3D object OB and the auxiliary image AU. The auxiliary image AU includes an image of the pointing area RG that has been drawn or otherwise processed by touch operation. For example, the auxiliary image AU is displayed behind the 3D object OB. The user US can see the auxiliary image AU by slightly shifting their gaze behind the 3D object OB during touch operation. Note that the position of the auxiliary image AU is not limited to behind the 3D object OB. The auxiliary image AU should be placed in a position that makes it easy for the user US to operate while comparing the input content.

[0054] [5. Specifying the pointing range for a remote 3D object] Figure 10 shows an example of specifying the pointing range RG for a remote 3D object OB.

[0055] If the 3D object OB is within the user US's reach, the user US can reach out and directly position the virtual board 50V in the vicinity of the 3D object OB. However, if the 3D object OB is beyond the user US's reach, such direct positioning is not possible. Therefore, the user US remotely manipulates the virtual board 50V displayed around the 3D object OB to position it in the vicinity of the 3D object OB.

[0056] For example, the video control unit 24 displays a virtual board 50V and a virtual input surface VI near a 3D object OB in conjunction with a remote selection operation of the 3D object OB. With this configuration, even remote 3D objects OB that are out of reach can be captured. The method of selection operation can be arbitrarily determined by the system developer. In the example in Figure 10, the selection operation is performed using a beam BM virtually displayed in the virtual space VS. The selection operation can be performed using the operating members (operation buttons, wheels, sticks, etc.) of the grip board 50.

[0057] The video control unit 24 adjusts the orientation of the virtual board 50V and virtual input surface VI to match the orientation of the grip board 50 and touch operation surface TC. The video control unit 24 displays the virtual board 50V and virtual input surface VI at a size corresponding to the distance between the user US and the 3D object OB. The greater the distance between the user US and the 3D object OB, the smaller the virtual board 50V and virtual input surface VI are displayed.

[0058] For example, when a user US initiates a selection operation by pressing a button, the video control unit 24 displays a virtual beam BM in the virtual space VS that extends forward from the user US's hand (for example, near the grip board 50). The video control unit 24 sets the direction of the beam BM based on the 6DoF position detected by the grip board 50. The user US then presses the button again with the beam BM shining on the desired 3D object OB. As a result, the 3D object OB shining on the beam BM is selected as the pointing target.

[0059] Unlike the example in Figure 4, the virtual board 50V and virtual input surface VI are displayed remotely. However, the method for setting the pointing range RG is the same as described above. That is, the user US changes the position and orientation of the remote virtual board 50V and virtual input surface VI by changing the position and orientation of the grip board 50 in their hand. The user US brings the virtual board 50V closer to the processing target area of ​​the 3D object OB and performs a capture operation. As a result, the area on the 3D object OB facing the virtual input surface VI is acquired as the pointing range RG.

[0060] Because remote 3D objects OB are displayed small, the actual processing area may be larger than the touch operation surface TC. If a wider area than the touch operation surface TC is to be acquired as the pointing range RG, the user US can enlarge the display size of the virtual board 50V and virtual input surface VI by operating buttons or other means.

[0061] Figure 11 shows how touch operations are performed on a remote 3D object OB.

[0062] When a capture operation is performed, the video control unit 24 hides the virtual board 50V and virtual input surface VI that were displayed near the 3D object OB. The video control unit 24 fixes the display position of the pointing range RG on the 3D object OB. The video control unit 24 displays the virtual pen 40V near the pointing range RG. The positional relationship between the virtual pen 40V and the pointing range RG is the same as the positional relationship between the touch pen 40 and the touch operation surface TC.

[0063] The display area DA for the 3D object OB and the user's work area WA are located in spatially separate locations. The display area DA refers to the area where the 3D object OB, pointing range RG, and virtual pen 40V are displayed. The work area WA refers to the area where the user US performs touch operations using the grip board 50 and the stylus 40. The user US performs touch operations on the remote pointing range RG in a relaxed posture.

[0064] [6. Modifications of the Grip Board] [6-1. Grip Board Integrated with Pick Mouth] The following describes modifications of the grip board. Figure 12 shows an example of a grip board 60 integrated with a pick mouth 10. The functional blocks of the grip board 60 are the same as those of the grip board 50 shown in Figure 3.

[0065] The grip board 60 comes in right-handed and left-handed versions. The right-handed version is designed so that the grip board 60 is held in the left hand and touch operations are performed with the right hand. The left-handed version is designed so that the grip board 60 is held in the right hand and touch operations are performed with the left hand. Figure 12 shows the right-handed grip board 60. The left-handed grip board 60 has a structure that is the reverse of Figure 12. The right-handed grip board 60 will be described below.

[0066] The grip board 60 has a board section BD and a grip section GP. The board section BD is a board-shaped input device having a touch-operable touch surface TC. The grip section GP is a handle attached to the back side of the board section BD (the side opposite to the touch surface TC). The grip section GP can be held by the user US who performs the touch operation. With this configuration, the board section BD is held stably. Therefore, the input work is stable.

[0067] The board section BD has an edge touch zone ET on the edge of the touch operation surface TC. The edge touch zone ET allows for touch operations separate from those performed on the touch operation surface TC. The board section BD also has a button zone BZ adjacent to the touch operation surface TC. The button zone BZ is provided with one or more operation buttons.

[0068] The board section BD has a palm rest PA. The palm rest PA is supported by the base of the thumb of the hand gripping the grip section GP. In the example shown in Figure 12, the palm rest PA is located on the opposite side of the touch operation surface TC from the button zone BZ. However, the position of the palm rest PA is not limited to this. Providing a wider palm rest PA increases the stability of touch operation.

[0069] The grip section GP includes a handle section HN and a pick mouse 10. The handle section HN is the part that the user US holds in their hand. The pick mouse 10 is a touch-operable pointing device. The pick mouse 10 is detachably connected to the handle section HN. This configuration provides a compact controller that integrates the board section BD, the handle section HN, and the pick mouse 10. The pick mouse 10, separated from the handle section HN, can be used in place of the stylus 40. The operating member OM of the pick mouse 10, integrated with the handle section HN, can be used for capture operations, etc.

[0070] One end of the handle section HN is connected to the back surface of the board section BD. The other end of the handle section HN is connected to the palm rest PA. The palm rest PA is supported by one end of the handle section HN. The board section BD is more stable when the handle section HN is held by the hand and the palm rest PA is further supported by the base of the thumb. Figure 13 is a five-view drawing of the grip board 60. Figure 14 is a drawing of the grip board 60 with the board section BD and grip section GP separated.

[0071] The left side of Figure 14 shows the pick mouse 10 separated from the handle HN. The pick mouse 10 functions as a pointing device for spatial manipulation within the XR system, having a physically contactable pen tip (pen tip PE). The pick mouse 10 is available in left-handed and right-handed versions. The pick mouse 10 in Figure 14 is the left-handed version. The left-handed and right-handed versions of the pick mouse 10 have a structure that is the reverse of each other.

[0072] For example, when the user holds the right-handed pick mouse 10 in their right hand and performs touch operations on the board BD, the left-handed pick mouse 10 is detachably connected to the handle HN. When the user holds the left-handed pick mouse 10 in their left hand and performs touch operations on the board BD, the right-handed pick mouse 10 is detachably connected to the handle HN.

[0073] The periphery of the touch surface TC has a different feel when touched compared to the touch surface TC itself. The touch sensation during touching allows the user to recognize the touch surface TC. Because the user's field of vision is obstructed by the HMD 20, it is difficult for the user to accurately recognize where they are performing a touch operation. If the inside and outside of the touch surface TC have different tactile sensations, the user can recognize whether the touch operation is being performed correctly within the touch surface TC based on the feel during touching.

[0074] Differences in tactile sensation can be created by providing differences in height or texture (such as material) between the touch surface TC and its surroundings. However, tactile identification may not always be sufficient. Therefore, the motion control unit 55 can notify the user US of a touch operation on the periphery of the touch surface TC with sound or vibration. This configuration allows the operator to recognize when a valid touch operation has not been performed.

[0075] Figure 15 shows how touch operations are performed using the pick mouse 10.

[0076] The pick mouse 10 has operating members OM (operating buttons, sticks, wheels, etc.) for performing various operations. The parts where operating members OM are not provided are grip surfaces that can be held by hand. The grip surface is about the size of a tennis ball and can be held in a way similar to holding a ball. The grip surface is provided with protrusions (finger rests FR) to stabilize the grip.

[0077] The pick mouse 10 has a pen tip PE, a wheel, a stick, and operating buttons as operating members OM. The function of each operating member OM can be arbitrarily set by the application. The types and number of operating members OM are not limited to those shown in Figure 12. The pen tip PE is used for pen input.

[0078] [6-2. Example of Grip Board Angle Adjustment] Figure 16 shows an example of angle adjustment of the grip board 60.

[0079] The angle at which the board portion BD feels comfortable for touch operation varies from person to person. The grip board 60 has a ball joint mechanism JT as an angle adjustment mechanism to accommodate such individual differences. The ball joint mechanism JT includes, for example, a ball stud BL and a fixing plate SC. The fixing plate SC is connected to the board portion BD by a rotating mounting mechanism such as a screw-in type. The ball stud BL is provided at the connection point between the fixing plate SC and the handle portion HN. The fixing plate SC has a socket (not shown) that encloses the ball stud BL and is rotatable in any direction. The rotational position of the socket can be fixed with a lever or the like.

[0080] [6-3. Example with multiple touch operation surfaces] Figure 17 shows an example with multiple touch operation surfaces TC.

[0081] When the area to be processed has a three-dimensional shape rather than a flat surface, it is preferable that the touch operation surface TC also has a three-dimensional structure that allows input from multiple directions. Figure 17 shows a touch operation surface TC that allows input from three directions. The three surfaces constituting the touch operation surface TC are arranged to be orthogonal to each other.

[0082] When performing touch operations on a three-dimensional object, it is sometimes necessary to sequentially perform touch operations while gradually shifting the pointing range RG to match the three-dimensional shape of the area to be processed. The pointing range RG can be changed by repeatedly performing the capture operation. However, if the touch operation surface TC has a structure that allows input from multiple directions, as in this example, the number of times the capture operation needs to be repeated is reduced, making the work easier.

[0083] [6-4. Touch operation surface having a pin array structure] Figure 18 shows a touch operation surface TC having a pin array structure AY.

[0084] If the surface shape of a 3D object OB can be realized on the touch operation surface TC, input and other operations can be performed with greater precision. Figure 18 shows a touch operation surface TC whose surface shape can be changed by a pin array structure AY.

[0085] The pin array structure AY has multiple pins PI whose protrusion amount can be individually controlled. The multiple pins PI are arranged two-dimensionally across the entire touch operation surface TC. The top of the pin array structure AY is covered with a cover sheet CS. The surface of the cover sheet CS has irregularities corresponding to the protrusion amount of each pin PI. Touch operations are performed on the cover sheet CS.

[0086] When the pointing range RG is acquired by the capture operation, the motion control unit 55 controls the protrusion amount of each pin PI based on the shape information of the 3D object OB that has been registered in advance. Depending on the protrusion amount of each pin PI, irregularities are formed on the surface of the cover sheet CS, and the surface shape of the 3D object OB of the pointing range RG is reproduced on the touch operation surface TC.

[0087] [6-5. Others] In the above-described embodiment, touch operations were performed using a pen-type controller such as a stylus 40 or a pick mouse 10. However, if the grip board 50 has a built-in XY detector capable of detecting capacitance, pressure, etc., the operating means may be a finger, or even just a stick (such as a brush) that does not interact electrically or magnetically with the grip board.

[0088] The grip board 50 has a structure similar to that of a regular tablet device. Therefore, in addition to pointing and inputting 3D objects OB as described in this disclosure, the grip board 50 can also perform general operations that are performed on tablet devices (such as a calculator, memo pad, menu display, and communication with other smartphones).

[0089] The method of holding the board section BD is not limited to gripping the grip section GP. Figures 19 and 20 show modified examples of the method of holding the board section BD. In the example in Figure 19, the hand is fixed to the board section BD by a back belt. The operating member OM is positioned at the location of the fingers of the fixed hand. In the example in Figure 20, the board section BD is held by grasping it with both hands from the left and right sides. The operating member OM is positioned on the side of the board section BD where the fingers are positioned.

[0090] [7. Modified Information Processing System] Figure 21 shows a modified version of the information processing system.

[0091] The information processing system 200 in this modified example differs from the information processing system 100 shown in Figure 2 in that a PC 30 is interposed between the stylus 40 and grip board 50 and the HMD 20. Information is exchanged between the stylus 40 and grip board 50 and the HMD 20 via the PC 30. The PC 30 includes a communication control unit 31, an operation control unit 32, a power supply control unit 33, an image control unit 34, and an input detection unit 35.

[0092] The communication control unit 31 acquires controller detection information from the stylus 40 and grip board 50, and acquires HMD detection information from the HMD 20. Based on the HMD detection information and controller detection information, the video control unit 34 can generate a portion of the video of the 3D object OB, or a portion of the video of the UI for pointing, moving, deforming, and processing the 3D object OB. The communication control unit 31 transmits the video generated by the video control unit 34 to the HMD 20.

[0093] The input detection unit 35 detects input made by the PC user to the PC 30. The operation control unit 32 controls the operation of the PC 30 based on the input operation to the PC 30. The power supply control unit 33 controls the power supply to the PC 30.

[0094] [8. Hardware Configuration Examples] Figure 22 shows examples of the hardware configurations of information processing systems 100 and 200.

[0095] The information processing systems 100 and 200 can be implemented by a computer 1000 as shown in Figure 22. The computer 1000 includes a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0096] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0097] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0098] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing systems 100 and 200 according to the embodiments of this disclosure, which are examples of program data 1450.

[0099] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0100] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0101] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device. The display unit 1700 corresponds to the display element of the HMD 20 or grip board 50 of this disclosure.

[0102] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0103] For example, when computer 1000 functions as an information processing system 100, 200 according to the embodiments of this disclosure, the processing circuit 1100 of computer 1000 functions as various detection and control units included in the information processing system 100, 200 by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program and various data according to this disclosure. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0104] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0105] [Note] The technology may also be configured as follows: (1) A controller having a board portion having a touch-operable touch surface, and an operation detection unit for detecting a capture operation to capture a 3D object that is the target of the touch operation in the vicinity of a virtual input surface whose position and orientation change in conjunction with changes in the position and orientation of the touch surface, while fixing the relative position and orientation between the 3D object and the virtual input surface. (2) The controller according to (1) above, having a grip portion attached to the side of the board portion opposite to the touch surface side, which can be held by the user performing the touch operation. (3) The controller according to (2) above, having a handle portion that the user holds in their hand, and a touch-operable pointing device detachably connected to the handle portion. (4) The controller according to (3) above, having a palm rest supported at one end of the handle portion. (5) The controller according to any one of (1) to (4) above, wherein the peripheral edge of the touch operation surface has a different feel from the touch operation surface when touched. (6) The controller according to any one of (1) to (5) above, wherein the peripheral edge of the touch operation surface has an action control unit that notifies that the touch operation has been performed. (7) An information processing system comprising: a controller having a touch operation surface that can be touched; and a video control unit that displays a virtual input surface near a 3D object that is the target of the touch operation, changes the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch operation surface, fixes the relative position and orientation of the virtual input surface and the 3D object based on a capture command from the controller, and changes the position and orientation of the 3D object in accordance with the movement of the arm holding the controller. (8) The information processing system according to (7) above, wherein the video control unit makes the orientation of the virtual input surface match the orientation of the touch operation surface.(9) The information processing system according to (7) or (8) above, wherein the video control unit displays the area on the 3D object obtained by projecting the virtual input surface onto the 3D object as the pointing area to be used for the touch operation. (10) The information processing system according to (9) above, wherein the video control unit displays a 2D image of the 3D object in parallel with the 3D display of the 3D object, showing the real-time operation status of the pointing area. (11) The information processing system according to (9) or (10) above, wherein the video control unit stops detecting and displaying the pointing area until it is determined that the virtual input surface is close to the 3D object based on a preset proximity condition. (12) The information processing system according to any one of (9) to (11) above, wherein the video control unit stops displaying the virtual input surface in conjunction with the acquisition of the capture command. (13) The information processing system according to any one of (7) to (12) above, wherein the video control unit displays the virtual input surface near the 3D object in conjunction with a remote selection operation of the 3D object. (14) An information processing method performed by a computer, comprising: displaying a virtual input surface near a 3D object that is the target of a touch operation; changing the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch operation surface of a controller; fixing the relative position and orientation of the virtual input surface and the 3D object based on a capture command from the controller; and changing the position and orientation of the 3D object in accordance with the movement of the arm holding the controller. (15) The information processing method according to (14) above, comprising making the orientation of the virtual input surface match the orientation of the touch operation surface. (16) The information processing method according to (14) or (15) above, comprising displaying the range on the 3D object obtained by projecting the virtual input surface onto the 3D object as the pointing range that is the target of the touch operation. (17) The information processing method according to (16) above, wherein, in parallel with the 3D display of the 3D object, a 2D image of the 3D object showing the real-time operation status to the pointing range is displayed.(18) The information processing method according to (16) or (17) above, comprising stopping the detection and display of the pointing range until the virtual input surface is deemed to be in close proximity to the 3D object based on a preset proximity condition. (19) The information processing method according to any one of (16) to (18) above, comprising stopping the display of the virtual input surface in conjunction with the acquisition of the capture command. (20) The information processing method according to any one of (14) to (19) above, comprising displaying the virtual input surface near the 3D object in conjunction with a remote selection operation of the 3D object.

[0106] 10 Pick Mouse (Pointing Device) 24 Video Control Unit 50 Grip Board (Controller) 53 Operation Detection Unit 55 Motion Control Unit 100, 200 Information Processing System BD Board Unit GP Grip Unit HN Handle Unit OB 3D Object PA Palm Rest RG Pointing Range TC Touch Operation Surface US User VI Virtual Input Surface

Claims

1. A controller comprising: a board portion having a touch-operable touch surface; and an operation detection unit for detecting a capture operation to capture a 3D object to be touched, in the vicinity of a virtual input surface whose position and orientation change in conjunction with changes in the position and orientation of the touch surface, while fixing the relative position and orientation between the 3D object and the virtual input surface.

2. The controller according to claim 1, wherein the controller has a grip portion attached to the side of the board portion opposite to the touch operation surface, and which can be held by the user performing the touch operation.

3. The controller according to claim 2, wherein the grip portion comprises a handle portion that the user holds in their hand, and a touch-operable pointing device detachably connected to the handle portion.

4. The controller according to claim 3, wherein the board portion has a palm rest supported at one end of the handle portion.

5. The controller according to claim 1, wherein the peripheral edge of the touch operation surface has a different feel from the touch operation surface when touched.

6. The controller according to claim 1, further comprising an operation control unit that notifies the peripheral edge of the touch operation surface that a touch operation has been performed.

7. An information processing system comprising: a controller having a touch-operable touch surface; and a video control unit that displays a virtual input surface near a 3D object to be touch-operated, changes the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch surface, fixes the relative position and orientation of the virtual input surface and the 3D object based on a capture command from the controller, and changes the position and orientation of the 3D object in accordance with the movement of the arm holding the controller.

8. The information processing system according to claim 7, wherein the video control unit makes the orientation of the virtual input surface match the orientation of the touch operation surface.

9. The information processing system according to claim 7, wherein the video control unit displays the area on the 3D object obtained by projecting the virtual input surface onto the 3D object as the pointing area to be used for the touch operation.

10. The information processing system according to claim 9, wherein the video control unit displays a 2D image of the 3D object in parallel with the 3D display of the 3D object, showing the real-time operation status of the pointing range.

11. The information processing system according to claim 9, wherein the video control unit stops detecting and displaying the pointing range until it is determined that the virtual input surface is in close proximity to the 3D object based on a preset proximity condition.

12. The information processing system according to claim 9, wherein the video control unit stops displaying the virtual input surface in conjunction with the acquisition of the capture command.

13. The information processing system according to claim 7, wherein the video control unit displays the virtual input surface near the 3D object in conjunction with a remote selection operation of the 3D object.

14. A computer-based information processing method comprising: displaying a virtual input surface near a 3D object that is the target of touch operation; changing the position and orientation of the virtual input surface in conjunction with changes in the position and orientation of the touch operation surface of a controller; fixing the relative position and orientation of the virtual input surface and the 3D object based on a capture command from the controller; and changing the position and orientation of the 3D object in accordance with the movement of the arm holding the controller.

15. The information processing method according to claim 14, comprising making the orientation of the virtual input surface match the orientation of the touch operation surface.

16. The information processing method according to claim 14, comprising projecting the virtual input surface onto the 3D object to display the area on the 3D object as the pointing area to be used for the touch operation.

17. The information processing method according to claim 16, comprising displaying a 2D image of the 3D object in parallel with the 3D display of the 3D object, showing the real-time operation status of the pointing range.

18. The information processing method according to claim 16, further comprising stopping the detection and display of the pointing range until the virtual input surface is deemed to be in close proximity to the 3D object based on a preset proximity condition.

19. The information processing method according to claim 16, further comprising stopping the display of the virtual input surface in conjunction with the acquisition of the capture command.

20. The information processing method according to claim 14, wherein the virtual input surface is displayed near the 3D object in conjunction with a remote selection operation of the 3D object.

Images