Controller, information processing system, and information processing method

Abstract

This controller comprises a 2D position detection unit, a grip surface, a finger hook unit, and a 6 DoF position detection unit. The 2D position detection unit detects a 2D position during an onboard operation. The grip surface is configured as a ball-shaped surface that can be gripped with a hand from above during the onboard operation. The finger hook unit is disposed in an interdigital space of a hand that grips the grip surface. The finger hook unit can stably hold the finger on the grip surface by being pressed against a side surface of the finger during gripping. The 6 DoF position detection unit detects a 6 DoF position during aerial operation.

Description

Controller, Information Processing System, and Information Processing Method

[0001] The present invention relates to a controller, an information processing system, and an information processing method.

[0002] In virtual space creation, currently, many tools are produced in a PC environment (e.g., 2D monitor + mouse: on-desk operation), and in the intermediate confirmation and integration stages, a virtual space environment (e.g., HMD + grip controller or hand interaction: in-air operation) is used. However, it is expected that the two production environments will be integrated in the future. In that case, as an efficient operation, a controller that can seamlessly transfer different events is required.

[0003] International Publication No. 2020 / 130356, US Patent Application Publication No. 2024 / 0248550, Japanese Unexamined Patent Application Publication No. 2008-047047, Japanese Unexamined Patent Application Publication No. 2009-238004, Japanese Unexamined Patent Application Publication No. 2013-025666, International Publication No. 2009 / 072583

[0004] As a conventional technique, a product with a gyro pointer function that lifts a mouse used on a desk into the air and operates a PC cursor is known. Various devices for enabling both on-desk operation and in-air operation are also disclosed in prior patents. However, in any example, stable operation cannot be performed without changing the gripping method between on-desk operation and in-air operation.

[0005] Therefore, the present disclosure proposes a controller, an information processing system, and an information processing method that can seamlessly perform on-desk operation and in-air operation without changing the gripping method.

[0006] According to the present disclosure, there is provided a controller including: a 2D position detection unit that detects a 2D position during on-desk operation; a spherical grip surface that can be held by covering the hand from above during the on-desk operation; a finger hook portion that is disposed in an interdigital cavity of a hand holding the grip surface and can stably hold the finger on the grip surface by being pressed against the side surface of the finger during gripping; and a 6DoF position detection unit that detects a 6DoF position during in-air operation.

[0007] According to this disclosure, an information processing system is provided, comprising: a communication control unit that acquires a 2D position obtained from a controller during desktop operation; and a video control unit that acquires a fixed position in a virtual space that is not linked to the movement of the controller, sets the direction of a beam emitted from the fixed position based on the 2D position, and controls the display position of a reference point on a 3D object emitted by the beam in accordance with the direction of the beam. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing system is performed by a computer.

[0008] This is a schematic diagram of a pick mouse, which is an example of a controller in this disclosure. This is a diagram showing an example of the configuration of the pick mouse. This is a diagram showing a six-view drawing of the pick mouse. This is a diagram showing the pick mouse held in the hand. This is a diagram showing an example of the arrangement of operation buttons. This is a diagram showing the switching of strokes by operating the buttons. This is a diagram showing a modified version of the operating member. This is a diagram showing an example of a size adjuster mechanism. This is a diagram showing an example of a sole surface adjustment mechanism. This is a diagram showing an example of how the pick mouse is used. This is a diagram showing how the pen tip is used as a pointer. This is a diagram showing an example of the configuration of an information processing system in which the pick mouse is applied as a controller. This is a diagram illustrating a pointing method using the pick mouse. This is a diagram showing an example of the display of the virtual space VS presented by the HMD. This is a diagram illustrating how to set a reference point in "desktop and 3D". This is a diagram showing a modified version of the information processing system. This is a diagram showing an example of the hardware configuration of the information processing system.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will proceed in the following order: [1. PickMouse] [2. Adjuster Mechanism] [3. Usage Forms of PickMouse] [4. Configuration of an Information Processing System Applying PickMouse] [5. Pointing Method Using PickMouse] [6. Modified Examples of Information Processing Systems] [7. Hardware Configuration Examples] [8. Effects]

[0011] [1. Pick Mouse] Figure 1 is a schematic diagram of a pick mouse 10, which is an example of a controller of this disclosure.

[0012] The PickMouse 10 is a controller that allows for seamless transitions between desktop and airborne operation without changing the grip. Desktop operation refers to pointing operations performed while scanning the controller on a support surface TB such as a desk. Airborne operation refers to pointing operations performed by lifting the controller with the hand HD and without support from a support surface TB.

[0013] Pointing operation refers to the input operation of position information corresponding to the controller's position in real space. The input position information is two-dimensional coordinate information (2D position) for desktop operations and three-dimensional coordinate information (6DoF position) for aerial operations. The support surface TB may be the surface of the desk, or a surface other than the desk, such as the floor or wall. The support surface TB may be a plane or a curved surface. Hereafter in this specification, the support surface TB may be referred to as "desktop".

[0014] The pick mouse 10 has operating components (operating buttons, sticks, wheels, etc.) for performing various operations. The portion without operating components is a grip surface GR (see Figure 2) that can be held by gripping it with the hand HD. The grip surface GR is about the size of a tennis ball and can be held in a way similar to gripping a ball.

[0015] The grip surface GR is provided with finger rests FR to stabilize the grip of the hand HD. The finger rests FR have one or more protrusions positioned in the interdigital spaces of the hand gripping the grip surface. The protrusions extend from the grip surface GR to a height approximately the same as the height of the fingers (for example, 1 cm to 3 cm). The user presses the fingers of the hand HD gripping the grip surface GR against the finger rests FR. This ensures that the fingers are stably held on the grip surface GR.

[0016] By providing a finger rest FR on the grip surface GR, the pick mouthpiece 10 can be held stably regardless of the grip position. Even when lifting the pick mouthpiece 10 from a desk-based operation position and changing the grip position in various ways, there is little risk of the pick mouthpiece 10 falling. Therefore, it is possible to seamlessly transition between desk-based operation and air-based operation without changing the grip.

[0017] Figure 2 shows an example of the configuration of the pick mouse 10. In the example in Figure 2, a right-handed pick mouse 10 is shown, which is held and operated with the right hand. Although not shown in the illustration, when the pick mouse 10 is held and operated with the left hand, a left-handed pick mouse 10 is used, which has the shape and arrangement of buttons etc. reversed from the example in Figure 2.

[0018] The pick mouse 10 has a pen tip PE, a wheel WH, a stick ST, and an operation button OP as operating members. The function of each operating member can be arbitrarily set by the application. The types and number of operating members are not limited to those shown in Figure 2. For example, the stick ST can be changed to a trackball TR (see Figure 5), etc. The pick mouse 10 has a grip surface GR in the part other than the part where the operating members are provided. The grip surface GR is configured as a ball-shaped surface that can be gripped by placing the hand HD over it when operating on a desk.

[0019] The pick mouthpiece 10 has finger rests FR on the grip surface GR. The finger rests FR are configured as protruding structures positioned in the interdigital spaces of the hand HD that grips the grip surface GR. In the example shown in Figure 2, the finger rests FR are configured as fin-shaped structures, but the structure of the finger rests FR is not limited to this. For example, the finger rests FR may be configured as columnar structures. The finger rests FR can stably hold the fingers on the grip surface GR by being pressed against the sides of the fingers when gripping.

[0020] Figure 3 shows six views of the pick mouthpiece 10. The top row of Figure 3 shows a top view. The bottom row of Figure 3 shows a bottom view. The four middle views of Figure 3, from left to right, show the right side view, front view, left side view, and rear view. Note that the direction covered by the hand HD when gripping is "upward," and the direction the fingertips of the hand HD are pointing is "forward" (front direction). Figure 4 shows the pick mouthpiece 10 in a hand HD grip.

[0021] The finger rest FR has one or more protrusions corresponding to different fingers. Each protrusion is pressed against the side of the corresponding finger when gripping, stabilizing the position and posture of the finger. The number of protrusions is not particularly limited. In the example in Figure 2, the finger rest FR has a middle finger protrusion PM and an index finger protrusion PI.

[0022] The middle finger projection PM is positioned between the index finger IF and middle finger MF of the hand HD gripping the grip surface GR. The right side of the middle finger projection PM becomes a support surface SM that supports the left side of the middle finger MF when gripping. The middle finger projection PM is pressed against the side of the middle finger MF when gripping, thereby stably holding the middle finger MF on the grip surface GR.

[0023] The index finger projection PI is positioned between the thumb TH and index finger IF of the hand HD gripping the grip surface GR. The right side of the index finger projection PI becomes a support surface SI that supports the left side of the index finger IF when gripping. The index finger projection PI is pressed against the side of the index finger IF when gripping, thereby stably holding the index finger IF on the grip surface GR.

[0024] The user holds the pick mouth 10 by gripping its left and right sides with their thumb TH and ring finger RF. The grip surface GR has an overhang shape at the position where the thumb TH and ring finger RF grip the grip surface GR (tightening position) to fit the gripping posture of the thumb TH and ring finger RF. An overhang shape means a shape where the tightening position is recessed, or a shape where the width above the tightening position is wider than the tightening position. By providing an overhang shape at the tightening position, the pick mouth 10 becomes less likely to fall during aerial operation.

[0025] For example, the left side of the pick mouth 10 has a recess where the fingertip of the thumb TH fits. This recess on the left side becomes the thumb holding surface HT, which is held by the thumb TH. The right side of the pick mouth 10 has a recess where the fingertip of the ring finger RF fits. This recess on the right side becomes the ring finger holding surface HR, which is held by the ring finger RF. The upper part of the ring finger holding surface HR is gently curved to follow the second joint of the ring finger RH. This curved upper part of the ring finger holding surface HR becomes the wrapping part WP, which the second joint of the ring finger RF wraps around.

[0026] The grip surface GR has a curved portion at the rear of the pick mouse that conforms to the shape of the palm of the hand (HD). This rear curved portion of the pick mouse 10 becomes the palm rest PR, which is held by the palm of the hand (HD). The palm rest PR fits the gripping posture of the palm of the hand (HD) that is holding the grip surface GR.

[0027] The bottom surface of the pick mouse 10 is a sole surface SL that contacts the support surface TB during desktop operation. A sensor SE for detecting 2D position is provided on the sole surface SL. An IR sensor, commonly used in mice and other devices, can be used as the sensor SE.

[0028] Figure 5 shows an example of the arrangement of the operation button OP.

[0029] The surface of the pick mouse 10 is provided with multiple operation buttons OP that can be operated with the fingertips of the hand HD holding the grip surface GR. In the examples of Figures 2 and 5, buttons BU1 to BU8 are provided as operation buttons OP. Multiple operation buttons OP (buttons BU3 to BU7 in the example of Figure 5) and the trackball TR provided on the left side of the pick mouse 10 are provided as a thumb operation section TO that can be operated with the thumb TH. The user can operate the thumb operation section TO while firmly gripping the grip surface GR with the index finger IF, middle finger MF, ring finger RF and palm.

[0030] The thumb operation section TO is provided within the range of motion of the thumb TH, with the position of the base of the thumb TH of the hand HD gripping the grip surface GR as the rotation center RC. For example, the range of motion of the thumb TH can be divided into multiple zones along the rotation direction of the thumb TH. One or more operating members (such as operation buttons OP or trackballs TR) are assigned to each zone.

[0031] In the example shown in Figure 5, the first, second, third, and fourth zones are provided sequentially from the top to the bottom of the left side of the pick mouse 10. Button BU3 is installed in the first zone. Trackball TR is installed in the second zone. Buttons BU4 and BU5 are installed in the third zone. Buttons BU6 and BU7 are installed in the fourth zone. The thumb resting surface HT is provided between the third and fourth zones.

[0032] The pick mouse 10 has buttons BU8, BU2, and a wheel WH within the range of motion of the index finger IF of the hand HD that grips the grip surface GR. The wheel WH is used for scrolling. Buttons BU8, BU2, and the wheel WH are provided as an index finger operation unit IO that can be operated by the index finger IF. The user can operate the index finger operation unit IO while firmly gripping the grip surface GR with the thumb TH, middle finger MF, ring finger RF, and palm.

[0033] Button BU2 is used, for example, in action games. In action games, there are scenes that require a large stroke (the amount of pressure applied to button BU2), such as when pulling a gun trigger or operating an accelerator. The size of the stroke can be switched using button BU8. The mechanism that activates button BU8 switches between a trigger state with a large stroke and a button state with a slight press. Figure 6 shows the stroke switching by operating button BU8. The left side of Figure 6 shows the trigger state. The right side of Figure 6 shows the button state.

[0034] A pen tip PE is located in front of the index finger interface (IF). The pen tip PE is used for pen input. The pen tip PE is positioned on a virtual plane that passes through the central axis of the index finger interface. The position indicated by the index finger interface intuitively coincides with the position of the tip of the pen tip PE. This minimizes the intuitive discrepancy in input position between touch input using the pen tip PE and touch input using the index finger interface.

[0035] The virtual plane passing through the central axis of the index finger IF is at a predetermined angle (ergonomic angle θ) with respect to the vertical plane. Ｅ It is tilted by only θ. Ergonomic angle θ Ｅ This is determined as the inclination angle of the hand (HD) when gripping the grip surface (GR) in a natural posture. Ergonomic angle θ Ｅ It is designed based on ergonomics.

[0036] The finger rest FR is shaped and sized such that the index finger operating section IO does not hit the ground regardless of the orientation in which the pick mouse 10 falls to the ground. For example, the height of the middle finger projection PM and the index finger projection PI is higher than the height of the buttons BU2, BU8, and the wheel WH. Therefore, the buttons BU2, BU8, and the wheel WH, which are positioned between the middle finger projection PM and the index finger projection PI, are less likely to be damaged by a fall.

[0037] Figure 7 shows a modified example of the operating member.

[0038] The type and number of operating elements can be arbitrarily designed by the device developer. In the example in Figure 7, a trackball TR and a double wheel DW are shown as alternative configurations to the stick ST. The stick ST is a lever-type angle input device that can be operated in any direction on two axes. The stick ST can return to the center by releasing the finger, and can also be pressed down.

[0039] The trackball TR is a ball-type angle input device that can be operated freely in two axes. The trackball TR can rotate endlessly but cannot be pushed in. The double wheel DW is an angle input device that can perform rotational operations in two orthogonal directions. The double wheel DW can rotate endlessly and can also be pushed in.

[0040] [2. Adjuster mechanism] FIGS. 8 and 9 are diagrams showing an example of the adjuster mechanism.

[0041] There are individual differences in the size of the hand HD that holds the grip surface GR. The pick mouse 10 has a size adjuster mechanism GJ as an adjuster mechanism corresponding to such individual differences. FIG. 8 is a diagram showing an example of the size adjuster mechanism GJ. The size adjuster mechanism GI can adjust the shape of the grip surface GR according to the size of the user's hand HD. In the example of FIG. 8, the grip surface GR of the part corresponding to the palm rest PR projects backward using a known mechanical mechanism. By adjusting the amount of projection according to the size of the palm, the feeling of fit when gripping is enhanced.

[0042] In the example of FIG. 8, the shape of the grip surface GR is adjusted by a mechanical mechanism provided inside the pick mouse 10. However, the shape of the grip surface GR can also be adjusted by an attachment attached to the grip surface GR. In the example of FIG. 8, the shape of the palm rest PR is adjusted, but the part to be adjusted is not limited to the palm rest PR. The pick mouse 10 may be provided with a size adjuster mechanism for adjusting the shape of other parts such as the winding part WP.

[0043] There are individual differences in how the pick mouse 10 is held. The pick mouse 10 has a sole surface adjustment mechanism SJ as an adjustment mechanism corresponding to such individual differences. FIG. 9 is a diagram showing an example of the sole surface adjustment mechanism SJ. The sole surface adjustment mechanism SJ can adjust the height or inclination of the sole surface SL grounded on the desktop (support surface TB). In the example of FIG. 9, the height and inclination of the pick mouse 10 are adjusted according to the user's preference by the attachment AT attached to the sole surface SL. The attachment AT can include a lens for adjusting the distance to the sensor SE.

[0044] [3. Usage form of pick mouse] FIG. 10 is a diagram showing an example of the usage form of the pick mouse 10.

[0045] The user US can perform touch input on the touch panel TP using the pen tip PE of the pick mouse 10. A pressure sensor may be built into the pen tip PE. Thereby, information processing based on the 2D position and pressure information can be realized. For example, control such as changing the thickness of characters according to the pen pressure becomes possible. In this case, by configuring the pen tip PE with a flexible material like a pen, input of pen characters and creation of watercolor paintings become possible. By making the pen tip PE of various materials detachable, the input variations can be increased. Note that the pressure sensor may be built into the touch panel TP instead of the pen tip PE.

[0046] The pen tip PE can also be used as a pointer indicating a position in the virtual space during an air operation. FIG. 11 is a diagram showing the usage form of the pen tip PE as a pointer. The relationship between the posture of the pick mouse 10 and the pointing direction (pointing direction) can be arbitrarily set by the system developer. The direction of the arm may be set as the pointing direction, or the direction of the pen tip PE may be set as the pointing direction.

[0047] The direction of the arm and the direction of the tip of the pen tip PE are at an angle θ ＰThe only difference is that when the tip of the pen tip PE is oriented in the pointing direction, operation can be performed in a natural and comfortable posture when pointing at a workspace close to the hand. However, when pointing at a workspace far away, the pick mouth 10 must be held with the wrist bent significantly backward, resulting in greater fatigue. Therefore, when pointing at a workspace close to the hand, the orientation of the pen tip PE is oriented in the pointing direction, and when pointing at a workspace far away, the orientation of the arm is oriented in the pointing direction.

[0048] [4. Configuration of an Information Processing System Using PickMouse] Figure 12 shows an example of the configuration of an information processing system 100 using the PickMouse 10 described above as a controller.

[0049] The information processing system 100 includes a pick mouse 10 and an HMD (Head Mounted Display) 20. The HMD 20 displays a 3D object OB in a virtual space VS (see Figure 13). The user US uses the pick mouse 10 to point to the 3D object OB. The user US performs processing such as inputting data to or processing the 3D object OB using a touch panel TP or the like as needed.

[0050] The pick mouse 10 includes a 2D position detection unit 11, a 6DoF position detection unit 12, an operation detection unit 13, a communication control unit 14, an operation control unit 15, and a power control unit 16. The 2D position detection unit 11 detects the 2D position during desktop operation. The 6DoF position detection unit 12 detects the 6DoF position during air operation. The operation detection unit 13 detects user operations on operating members (operation buttons OP, stick ST, trackball TR, wheel WH, etc.).

[0051] The motion control unit 15 acquires the 2D position, 6DoF position, and operation information of the operating member acquired by the pick mouse 10 as controller detection information. The motion control unit 15 controls the operation of the pick mouse 10 based on the controller detection information. The communication control unit 14 outputs the controller detection information to the HMD 20. The power supply control unit 16 controls the power supply of the pick mouse 10.

[0052] 2D position detection is performed using sensors SE such as IR sensors provided on the sole surface SL. 6DoF position detection is performed using information from cameras and inertial sensors built into the pick mouse 10. Techniques such as SLAM (Simultaneous Localization and Mapping) can be used for the calculation method. Information from cameras and inertial sensors may also be used for 2D position detection.

[0053] The HMD 20 includes a communication control unit 21, an operation control unit 22, a power supply control unit 23, an image control unit 24, an audio control unit 25, an operation detection unit 26, and a controller position detection unit 27. The communication control unit 21 acquires controller detection information from the pick mouse 10. The controller detection information includes the 2D position obtained during desktop operation, the 6DoF position obtained during aerial operation, and operation information of various operating members.

[0054] The controller position detection unit 27 detects the position information of the HMD 20 as the controller position. The controller position is detected using information from the camera and inertial sensors built into the HMD 20. Technologies such as SLAM can be used for the calculation method. The operation detection unit 26 detects the user's (US) operation on the operating member built into the HMD 20.

[0055] The video control unit 24 acquires the position information of the HMD 20 and the user's (US) operation information on the HMD 20 as HMD detection information. Based on the HMD detection information, the video control unit 24 generates an image of the 3D object OB. Based on the HMD detection information and the controller detection information, the video control unit 24 generates an image of the UI (User Interface) for pointing, moving, deforming, and processing the 3D object OB. The video control unit 24 displays the generated image on a display screen using a light source such as an SLD (Superluminescent diode).

[0056] The audio control unit 25 outputs audio in accordance with the video. The motion control unit 22 controls the operation of the HMD 20 based on HMD detection information. The power supply control unit 23 controls the power supply of the HMD 20.

[0057] [5. Pointing Method Using a Pick Mouse] Figure 13 is a diagram illustrating the pointing method using a pick mouse 10.

[0058] The HMD 20 displays the 3D object OB as either a 2D or 3D image. The pick mouse 10 points to the input position (reference point RP) through desktop and airborne operations. The user US can arbitrarily combine the display mode of the 3D object OB (2D image, 3D image) and the operation mode of the pick mouse 10 (desktop operation, airborne operation).

[0059] For example, in Figure 13, "Desktop / Plane" indicates pointing to a 3D object OB displayed in a 2D image using desktop operations. "Desktop / 3D" indicates pointing to a 3D object OB displayed in a 3D image using desktop operations. "Air / Plane" indicates pointing to a 3D object OB displayed in a 2D image using air operations. "Air / 3D" indicates pointing to a 3D object OB displayed in a 3D image using air operations.

[0060] Figure 14 shows an example of the display of the virtual space VS presented by the HMD 20.

[0061] The video control unit 24 displays a 3D object OB in the virtual space VS. The movement of the pick mouse 10 can be recognized by the movement of the virtual controller 10V. The virtual controller 10V is a 3D object in the virtual space VS that mimics the pick mouse 10. The virtual controller 10V exhibits the same movement as the pick mouse 10. The video control unit 24 displays the virtual controller 10V at a position in the virtual space VS that corresponds to the position of the pick mouse 10 in real space.

[0062] "On a desk / plane" is equivalent to normal mouse operation. "In the air / 3D" and "In the air / plane" are operated in the same way as the operation of a controller for a normal XR device described in Patent Document 1, etc. For example, the user US changes the position of the reference point RP by changing the orientation of the pick mouse 10. To make the position of the reference point RP easier to understand, a virtual beam BM is displayed in the virtual space VS, pointing from the virtual controller 10V (for example, the part corresponding to the tip of the pen tip PE) to the reference point RP.

[0063] "Desktop / 3D" converts the movement of the pick mouse 10 on the support surface TB into the three-dimensional movement of the reference point RP. This process includes mapping the two-dimensional movement (2D position) of the pick mouse 10 into 3D space. Conventionally, there have been no examples of such processing being performed with mouse operations. Therefore, "Desktop / 3D" is a novel operation unlike anything seen before. "Desktop / 3D" will be explained in detail below.

[0064] Figure 15 illustrates how to set the reference point RP in "desktop and 3D" models.

[0065] The video control unit 24 displays a beam BM for pointing in the virtual space VS. The beam BM is illuminated from a fixed position (light source LS) in the virtual space VS. "Fixed" means that the position does not change in conjunction with the movement of the pick mouse 10. The video control unit 24 acquires the position in the virtual space VS that is not linked to the movement of the pick mouse 10 as the fixed position. The video control unit 24 sets the orientation of the beam BM based on the 2D position detected by the pick mouse 10. The video control unit 24 controls the display position of the reference point RP on the 3D object OB illuminated by the beam BM to match the orientation of the beam BM (see the upper diagram in Figure 15).

[0066] For example, the video control unit 24 sets a sphere centered on the light source LS of the beam BM as the coordinate sphere within the virtual space VS. The video control unit 24 sets the intersection point of the coordinate sphere and the beam BM as the relay point RL. The video control unit 24 sets the direction in which the line of intersection between the coordinate sphere and the horizontal plane extends as the X-axis, and sets the direction in which the line of intersection between the coordinate sphere and the vertical plane extends as the Y-direction.

[0067] The video control unit 24 sets up a 2D coordinate system on the support surface TB with the position of the pick mouse 10 as the origin. For example, in the 2D coordinate system, the direction in which the tip of the pen tip PE points is the y-direction, and the direction perpendicular to the y-direction is the x-direction. The video control unit 24 acquires the x and y coordinates of the pick mouse 10 in the 2D coordinate system as the 2D position.

[0068] The video control unit 24 converts the movement of the pick mouse 10 in the 2D coordinate system into the movement of the relay point RL in the XY coordinate system. For example, if the position of the pick mouse 10 in the 2D coordinate system is (x, y) and the position of the relay point RL in the XY coordinate system is (X, Y), the video control unit 24 sets X = a × x and Y = b × y. "a" and "b" represent constants. The values ​​of "a" and "b" can be arbitrarily set by the system developer.

[0069] The video control unit 24 changes the direction of the beam BM by moving the position of the relay point RL according to the 2D position of the pick mouse 10. The video control unit 24 acquires a point on the surface of the 3D object OB where the beam BM intersects as a reference point RP.

[0070] The position of the light source LS of the beam BM can be arbitrarily set by the system developer. Considering the visibility of the beam BM, it is preferable to set the light source LS in a position that does not cause strong congestion for the user US. For example, the video control unit 24 can set the position of the light source LS of the beam BM in desktop operation to a position different from the viewpoint of the user US operating the pick mouse 10.

[0071] If the light source LS is set to a position significantly offset from the user's (US) viewpoint, the positions of the beam BM and reference point RP become difficult to recognize. Therefore, it is preferable to set the light source LS near the user's (US) head or torso. For example, the video control unit 24 can set the user's (US) neck, torso, arms, or top of head as the position of the light source LS for the beam BM in desktop operation.

[0072] The user US can lift the pick mouse 10 without changing how they hold it and perform aerial operations. Switching between operation modes is detected based on button operation by the user US or changes in the posture of the pick mouse 10 detected from sensor information. During aerial operations, the light source LS of the beam BM is fixed to the position of the tip of the pen tip PE, and the direction of the beam BM is set to the direction of the tip of the pen tip PE. The position of the reference point RP is controlled based on the 6DoF position detected by the pick mouse 10.

[0073] For example, the communication control unit 21 obtains mode switching information from the pick mouse 10, indicating that the operation mode has been switched from desktop operation to aerial operation, and the 6DoF position obtained by the aerial operation. The video control unit 24 sets the position of the light source LS of the beam BM and the orientation of the beam BM based on the 6DoF position. The video control unit 24 controls the display position of the reference point RP to match the position of the light source LS of the beam BM and the orientation of the beam BM.

[0074] When transitioning from desktop operation to airborne operation, the position of the reference point RP shifts due to changes in the position and orientation of the pick mouse 10. To avoid the shift of the reference point RP when switching operation modes, the pick mouse 10 can be provided with an operation button OP (reference point fixing button) to fix the reference point RP. The installation position of the reference point fixing button is arbitrary. It is preferable to install the reference point fixing button on the thumb operation part TO or the index finger operation part IO, as this stabilizes the grip of the pick mouse 10 when the button is operated (pressed down on the reference point fixing button).

[0075] For example, the thumb control unit TO has a reference point lock button. The reference point lock button outputs an instruction that prohibits the position of the reference point RP, which is set in the virtual space VS, from being changed by switching between operation modes between desktop operation and air operation, and by changes in 2D position and 6DoF position. The communication control unit 21 receives a prohibition instruction from the pick mouse 10 indicating that the change in the position of the reference point RP is prohibited. The video control unit 24 locks the position of the reference point RP from the time it receives the prohibition instruction until it receives a release instruction from the user US (see the lower left diagram in Figure 15).

[0076] After receiving a prohibition instruction, the communication control unit 21 acquires mode switching information indicating a switch from desktop operation to air operation. The video control unit 24 switches the position of the light source LS of the beam BM to the position of the tip of the pen tip PE. Until a release instruction is received from the user US, the video control unit 24 controls the display of the beam BM so that the beam BM is projected from the position of the light source LS corresponding to the 6DoF position toward the position of the fixed reference point RP (see the lower right diagram of Figure 15).

[0077] The communication control unit 21 acquires mode switching information indicating a switch from desktop operation to airborne operation, and then acquires a release instruction from the user US. The method of issuing the release instruction can be arbitrarily set by the system developer. For example, the user US can issue a release instruction by releasing the press of the reference point fixing button, which was pressed to fix the reference point RP.

[0078] When the video control unit 24 receives a release command, it sets the position of the light source LS of the beam BM and the orientation of the beam BM based on the 6DoF position. The video control unit 24 controls the display position of the reference point RP to match the position of the light source LS of the beam BM and the orientation of the beam BM (see the lower right diagram in Figure 13). The video control unit 24 can make the display patterns (color, shape, etc.) of the beam BM and the virtual controller 10V different for each operation mode so that the user US can recognize the current operation mode.

[0079] In the above explanation, an example was given in which a dedicated operation button OP for issuing a prohibition command is provided on the pick mouse 10. However, prohibition commands do not necessarily have to be issued by a dedicated operation button OP. For example, the video control unit 24 can acquire a reference point RP determination operation by the pick mouse 10 as a prohibition command. A reference point RP determination operation means an operation that confirms the current reference point RP as the target for processing. The video control unit 24 can also acquire input or determination operations from controllers other than the pick mouse 10 (such as other controllers held in the hand opposite to the hand holding the pick mouse 10, or a keyboard) as prohibition commands.

[0080] [6. Modified Information Processing System] Figure 16 shows a modified version of the information processing system.

[0081] The information processing system 200 in this modified example differs from the information processing system 100 shown in Figure 12 in that a PC 30 is interposed between the pick mouse 10 and the HMD 20. Information is exchanged between the pick mouse 10 and the HMD 20 via the PC 30. The PC 30 includes a communication control unit 31, an operation control unit 32, a power supply control unit 33, an image control unit 34, and an input detection unit 35.

[0082] The communication control unit 31 acquires controller detection information from the pick mouse 10 and HMD detection information from the HMD 20. Based on the HMD detection information and controller detection information, the video control unit 34 can generate a portion of the video of the 3D object OB, or a portion of the video of the UI for pointing, moving, deforming, and processing the 3D object OB. The communication control unit 31 transmits the video generated by the video control unit 34 to the HMD 20.

[0083] The input detection unit 35 detects input made by the PC user to the PC 30. The operation control unit 32 controls the operation of the PC 30 based on the input operation to the PC 30. The power supply control unit 33 controls the power supply to the PC 30.

[0084] [7. Hardware Configuration Examples] Figure 17 shows examples of the hardware configurations of information processing systems 100 and 200.

[0085] The information processing systems 100 and 200 can be implemented by a computer 1000 as shown in Figure 17. The computer 1000 includes a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of the computer 1000 are connected by a bus 1050.

[0086] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0087] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0088] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing systems 100 and 200 according to the embodiments of this disclosure, which are examples of program data 1450.

[0089] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0090] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0091] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device. The display unit 1700 corresponds to the display element of the HMD 20 or the like in this disclosure.

[0092] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0093] For example, when computer 1000 functions as an information processing system 100, 200 according to the embodiments of this disclosure, the processing circuit 1100 of computer 1000 functions as various detection and control units included in the information processing system 100, 200 by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program and various data according to this disclosure. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0094] [8. Effects] The pick mouse 10 has a 2D position detection unit 11, a grip surface GR, a finger rest FR, and a 6DoF position detection unit 12. The 2D position detection unit 11 detects the 2D position during desktop operation. The grip surface GR is configured as a ball-shaped surface that can be gripped by placing the hand HD over it during desktop operation. The finger rest FR is positioned in the interdigital space of the hand HD that grips the grip surface GR. The finger rest FR is pressed against the side of the fingers during gripping, allowing the fingers to be stably held on the grip surface GR. The 6DoF position detection unit 12 detects the 6DoF position during air operation.

[0095] This configuration allows the pick mouse 10 to be held stably for both desktop and airborne operations using the same hand grip. Therefore, desktop and airborne operations can be performed seamlessly without changing the gripping method.

[0096] The finger rest FR has a projection PM for the middle finger. The projection PM is positioned between the index finger IF and middle finger MF of the hand HD that grips the grip surface GR. When gripping, the projection PM is pressed against the side of the middle finger MF, thereby stably holding the middle finger MF on the grip surface GR.

[0097] This configuration increases stability when gripping the handle.

[0098] The finger rest FR has a projection PI for the index finger. The projection PI is positioned between the thumb TH and the index finger IF of the hand HD that grips the grip surface GR. When gripping, the projection PI is pressed against the side of the index finger IF, thereby stably holding the index finger IF on the grip surface GR.

[0099] This configuration increases stability when gripping the handle.

[0100] The grip surface GR has an overhang shape that fits the gripping posture of the thumb TH and ring finger RF, at the position where the grip surface GR is sandwiched between the thumb TH and ring finger RF.

[0101] This configuration increases stability when gripping the handle.

[0102] The grip surface GR has a palm rest PR that fits the gripping position of the palm of the hand holding the grip surface GR.

[0103] This configuration increases stability when gripping the handle.

[0104] The pick mouthpiece 10 has a size adjuster mechanism GJ. The size adjuster mechanism GJ allows the shape of the grip surface GR to be adjusted to match the size of the user's hand HD.

[0105] This configuration allows for adjustment of the grip surface (GR) to suit individual differences.

[0106] The pick mouse 10 has a sole surface adjustment mechanism SJ. The sole surface adjustment mechanism SJ can adjust the height or inclination of the sole surface SL that is in contact with the desk.

[0107] This configuration allows users (US) to adjust the sole surface (SL) to their preference, thereby improving usability when using the device on a desk.

[0108] The pick mouth 10 has a thumb operating part TO that can be operated by the thumb TH. The thumb operating part TO is positioned within the range of motion of the thumb TH, with the position of the base of the thumb TH of the hand HD that grips the grip surface GR as the rotation center RC.

[0109] This configuration allows for operation using the thumb (TH) while stably holding the grip surface (GR).

[0110] The thumb control unit TO has an operation button OP. The operation button OP outputs an instruction that prohibits the position of the reference point RP set in the virtual space VS from being changed by (A) switching between operation modes between desktop operation and air operation, and (B) changes in 2D position and 6DoF position.

[0111] With this configuration, the position of the reference point RP does not shift even if the orientation of the pick mouse 10 changes due to a switch in the operating mode.

[0112] The pick mouse 10 has an index finger operation section IO that can be operated by the index finger IF. The index finger operation section IO is positioned within the range of motion of the index finger IF of the hand HD that grips the grip surface GR.

[0113] This configuration allows for stable gripping of the grip surface (GR) while performing operations with the index finger (IF).

[0114] The information processing systems 100 and 200 include a communication control unit 21 and a video control unit 24. The communication control unit 21 acquires a 2D position obtained from the pick mouse 10 during desktop operation. The video control unit 24 acquires a fixed position in a virtual space VS that is not linked to the movement of the pick mouse 10. Based on the 2D position, the video control unit 24 sets the direction of the beam BM irradiated from the fixed position. The video control unit 24 controls the display position of the reference point RP on the 3D object OB irradiated by the beam BM to match the direction of the beam BM. In the information processing method of this disclosure, the processing of the information processing systems 100 and 200 is executed by a computer.

[0115] In this configuration, the light source for the beam BM is set independently of the pick mouse 10. The beam BM, which is at a different height from the support surface TB, can be stably scanned by tabletop operation. Compared to scanning the beam BM by aerial operation without support from the support surface TB, the position of the reference point RP can be set with greater precision.

[0116] The communication control unit 21 receives mode switching information from the pick mouse 10, indicating that the operation mode has been switched from desktop operation to aerial operation, and the 6DoF position obtained by the aerial operation. The video control unit 24 sets the position of the light source LS of the beam BM and the orientation of the beam BM based on the 6DoF position. The video control unit 24 controls the display position of the reference point RP to match the position of the light source LS of the beam BM and the orientation of the beam BM.

[0117] This configuration allows for the setting of the reference point RP through aerial manipulation.

[0118] The communication control unit 21 receives a prohibition instruction from the pick mouse 10 indicating that the position of the reference point RP should not be changed. The video control unit 24 fixes the position of the reference point RP from the time it receives the prohibition instruction until it receives a release instruction from the user US.

[0119] With this configuration, the position of the reference point RP does not shift even if the orientation of the pick mouse 10 changes due to a switch in the operating mode.

[0120] The video control unit 24 receives the operation to determine the reference point RP by the pick mouse 10 as a prohibition instruction.

[0121] With this configuration, the position of the reference point RP at the time of decision is maintained without the need for any special operations to issue a prohibition instruction.

[0122] After receiving a prohibition instruction, the communication control unit 21 acquires mode switching information indicating a switch from desktop operation to aerial operation. The video control unit 24 controls the display of beam BM so that beam BM is directed from the position of the light source LS corresponding to the 6DoF position toward the position of the fixed reference point RP until a release instruction is received from the user US.

[0123] With this configuration, while confirming the position of the beam BM, the posture of the hand HD operating the pick mouse 10 can be changed to a position that facilitates aerial operation.

[0124] The communication control unit 21 acquires mode switching information indicating a switch from desktop operation to aerial operation, and then receives a release instruction from the user US. The video control unit 24 sets the position of the light source LS of the beam BM and the orientation of the beam BM based on the 6DoF position. The video control unit 24 controls the display position of the reference point RP to match the position of the light source LS of the beam BM and the orientation of the beam BM.

[0125] This configuration allows for the setting of the reference point RP through aerial manipulation.

[0126] The video control unit 24 displays the virtual controller 10V at a position in the virtual space VS that corresponds to the position of the pick mouse 10 in real space.

[0127] With this configuration, you can intuitively operate the system while observing the movement of the virtual controller 10V.

[0128] The video control unit 24 makes the display modes of the beam BM and virtual controller 10V different for each operation mode.

[0129] This configuration allows for clear recognition of the currently running operating mode.

[0130] The video control unit 24 sets the position of the light source LS of the beam BM in desktop operation to a position different from the viewpoint of the user US operating the pick mouse 10.

[0131] This configuration makes the beam BM easier to see.

[0132] The video control unit 24 sets the user US's neck, torso, arms, or top of head as the position of the light source LS of the beam BM during desktop operation.

[0133] This configuration makes it easier to determine the position of the beam BM and to operate the beam BM.

[0134] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0135] [Note] The technology can also be configured as follows: (1) A controller comprising: a 2D position detection unit for detecting the 2D position during desktop operation; a ball-shaped grip surface that can be gripped by placing a hand over it from above during desktop operation; a finger rest portion positioned in the interdigital space of the hand gripping the grip surface, which is pressed against the side of the fingers when gripping to stably hold the fingers on the grip surface; and a 6DoF position detection unit for detecting the 6DoF position during aerial operation. (2) The controller according to (1) above, wherein the finger rest portion has a projection for the middle finger positioned between the index finger and middle finger of the hand gripping the grip surface, and the projection for the middle finger is pressed against the side of the middle finger when gripping to stably hold the middle finger on the grip surface. (3) The controller according to (1) or (2) above, wherein the finger rest portion has an index finger projection positioned between the thumb and index finger of the hand gripping the grip surface, and the index finger projection is pressed against the side of the index finger when gripping to stably hold the index finger on the grip surface. (4) The controller according to any one of (1) to (3) above, wherein the grip surface has an overhang shape that fits the gripping posture of the thumb and ring finger at a position where the grip surface is held between the thumb and ring finger. (5) The controller according to any one of (1) to (4) above, wherein the grip surface has a palm rest that fits the gripping posture of the palm of the hand gripping the grip surface. (6) The controller according to any one of (1) to (5) above, which has a size adjuster mechanism that can adjust the shape of the grip surface to match the size of the user's hand. (7) A controller according to any one of (1) to (6) above, having a sole surface adjustment mechanism that can adjust the height or inclination of the sole surface that makes contact with the desk. (8) A controller according to any one of (1) to (7) above, having a thumb operation part that can be operated by the thumb, within the range of motion of the thumb with the position of the base of the thumb of the hand that grips the grip surface as the rotation center.(9) The controller according to (8) above, wherein the thumb operation unit has an operation button that outputs an instruction prohibiting the position of a reference point set in the virtual space from being changed by switching between operation modes between the desktop operation and the air operation, and by fluctuations in the 2D position and the 6DoF position. (10) The controller according to any one of (1) to (9) above, wherein the index finger operation unit is within the range of motion of the index finger of the hand that grips the grip surface and is operable by the index finger. (11) An information processing system having: a communication control unit that acquires a 2D position obtained in desktop operation from the controller; and a video control unit that acquires a fixed position in the virtual space that is not linked to the movement of the controller, sets the direction of a beam emitted from the fixed position based on the 2D position, and controls the display position of a reference point on a 3D object emitted by the beam in accordance with the direction of the beam. (12) The information processing system according to (11), wherein the communication control unit receives from the controller mode switching information indicating that the operation mode has been switched from desktop operation to aerial operation, and the 6DoF position obtained by the aerial operation, and the video control unit sets the position of the light source of the beam and the direction of the beam based on the 6DoF position, and controls the display position of the reference point to match the position of the light source of the beam and the direction of the beam. (13) The information processing system according to (12), wherein the communication control unit receives from the controller a prohibition instruction indicating that the change of the position of the reference point is prohibited, and the video control unit fixes the position of the reference point from the time it receives the prohibition instruction until it receives a release instruction from the user. (14) The information processing system according to (13), wherein the video control unit receives the controller's operation to determine the reference point as the prohibition instruction.(15) The information processing system according to (13) or (14) above, wherein the communication control unit, after obtaining the prohibition instruction, obtains the mode switching information indicating a switch from the desktop operation to the air operation, and the video control unit controls the display of the beam so that the beam is irradiated from the position of the light source corresponding to the 6DoF position toward the position of the fixed reference point until the release instruction is obtained from the user. (16) The information processing system according to (15) above, wherein the communication control unit, after obtaining the mode switching information indicating a switch from the desktop operation to the air operation, obtains the release instruction from the user, and the video control unit sets the position of the light source and the direction of the beam based on the 6DoF position, and controls the display position of the reference point in accordance with the position of the light source and the direction of the beam. (17) The information processing system according to any one of (12) to (16) above, wherein the video control unit displays a virtual controller at the position in the virtual space corresponding to the position of the controller in the real space. (18) The information processing system according to (17) above, wherein the video control unit causes the display of the beam and the virtual controller to differ for each operation mode. (19) The information processing system according to any one of (11) to (18) above, wherein the video control unit sets the position of the light source of the beam in the desktop operation to a position different from the viewpoint of the user operating the controller. (20) The information processing system according to (19) above, wherein the video control unit sets the position of the light source of the beam in the desktop operation to the user's neck, torso, arms, or top of head. (21) An information processing method performed by a computer, comprising: acquiring a 2D position obtained from the controller in desktop operation; acquiring a fixed position in a virtual space that is not linked to the movement of the controller; setting the direction of the beam irradiated from the fixed position based on the 2D position; and controlling the display position of a reference point on a 3D object irradiated by the beam to match the direction of the beam.

[0136] 10 Pick Mouse (Controller) 10V Virtual Controller 11 2D Position Detection Unit 12 6DoF Position Detection Unit 21 Communication Control Unit 24 Video Control Unit BM Beam FR Finger Rest GJ Size Adjuster Mechanism GR Grip Surface HD Hand IF Index Finger IO Index Finger Operation Unit LS Light Source MF Middle Finger OB 3D Object OP Operation Button PI Protrusion for Index Finger PM Protrusion for Middle Finger PR Palm Rest RF Ring Finger RP Reference Point SJ Sole Surface Adjustment Mechanism TH Thumb TO Thumb Operation Unit US User VS Virtual Space

Claims

1. A controller comprising: a 2D position detection unit for detecting the 2D position during desktop operation; a ball-shaped grip surface that can be gripped by placing a hand over it from above during desktop operation; a finger rest portion positioned in the interdigital space of the hand gripping the grip surface, which is pressed against the sides of the fingers during gripping to stably hold the fingers on the grip surface; and a 6DoF position detection unit for detecting the 6DoF position during airborne operation.

2. The controller according to claim 1, wherein the finger rest portion has a projection for the middle finger positioned between the index finger and middle finger of the hand that grips the grip surface, and the projection for the middle finger is pressed against the side of the middle finger when gripping, thereby stably holding the middle finger on the grip surface.

3. The controller according to claim 1, wherein the finger rest portion has an index finger projection positioned between the thumb and index finger of the hand gripping the grip surface, and the index finger projection is pressed against the side of the index finger when gripping, thereby stably holding the index finger on the grip surface.

4. The controller according to claim 1, wherein the grip surface has an overhang shape that fits the gripping posture of the thumb and ring finger at a position where the grip surface is held between the thumb and ring finger.

5. The controller according to claim 1, wherein the grip surface has a palm rest that fits the gripping posture of the palm of the hand holding the grip surface.

6. The controller according to claim 1, further comprising a size adjuster mechanism capable of adjusting the shape of the grip surface to match the size of the user's hand.

7. The controller according to claim 1, having a sole surface adjustment mechanism that can adjust the height or inclination of the sole surface that makes contact with the desk.

8. The controller according to claim 1, further comprising a thumb operating section that can be operated by the thumb, within the range of motion of the thumb, with the position of the base of the thumb of the hand gripping the grip surface as the rotational center.

9. The controller according to claim 8, wherein the thumb operation unit has an operation button that outputs an instruction prohibiting the position of a reference point set in the virtual space from being changed by switching between operation modes between the desktop operation and the air operation, and by fluctuations in the 2D position and the 6DoF position.

10. The controller according to claim 1, further comprising an index finger operating section that can be operated by the index finger, within the range of motion of the index finger of the hand gripping the grip surface.

11. An information processing system comprising: a communication control unit that acquires a 2D position obtained from a controller during desktop operation; and a video control unit that acquires a fixed position in a virtual space that is not linked to the movement of the controller, sets the direction of a beam emitted from the fixed position based on the 2D position, and controls the display position of a reference point on a 3D object emitted by the beam in accordance with the direction of the beam.

12. The information processing system according to claim 11, wherein the communication control unit obtains from the controller mode switching information indicating that the operation mode has been switched from desktop operation to aerial operation, and the 6DoF position obtained by the aerial operation, and the video control unit sets the position of the light source of the beam and the direction of the beam based on the 6DoF position, and controls the display position of the reference point in accordance with the position of the light source of the beam and the direction of the beam.

13. The information processing system according to claim 12, wherein the communication control unit receives a prohibition instruction from the controller indicating that the change of the position of the reference point is prohibited, and the video control unit fixes the position of the reference point from the time it receives the prohibition instruction until it receives a release instruction from the user.

14. The information processing system according to claim 13, wherein the video control unit acquires the controller's operation to determine the reference point as the prohibition instruction.

15. The information processing system according to claim 13, wherein the communication control unit, after receiving the prohibition instruction, acquires the mode switching information indicating a switch from the desktop operation to the aerial operation, and the video control unit controls the display of the beam so that the beam is irradiated from the position of the light source corresponding to the 6DoF position toward the position of the fixed reference point until the release instruction is received from the user.

16. The information processing system according to claim 15, wherein the communication control unit, after acquiring the mode switching information indicating a switch from the desktop operation to the aerial operation, acquires the release instruction from the user, and the video control unit sets the position of the beam's light source and the direction of the beam based on the 6DoF position, and controls the display position of the reference point in accordance with the position of the beam's light source and the direction of the beam.

17. The information processing system according to claim 12, wherein the video control unit displays a virtual controller at a position in the virtual space corresponding to the position of the controller in the real space.

18. The information processing system according to claim 17, wherein the video control unit causes the display of the beam and the virtual controller to differ for each operation mode.

19. The information processing system according to claim 11, wherein the video control unit sets the position of the light source of the beam in the desktop operation to a position different from the viewpoint of the user operating the controller.

20. The information processing system according to claim 19, wherein the video control unit sets the user's neck, torso, arms, or top of head as the position of the light source for the beam in the desk operation.

21. A computer-based information processing method comprising: acquiring a 2D position obtained from a controller during desktop operation; acquiring a fixed position in a virtual space that is not linked to the movement of the controller; setting the direction of a beam emitted from the fixed position based on the 2D position; and controlling the display position of a reference point on a 3D object illuminated by the beam to match the direction of the beam.

Images