Information processing device, information processing system, information processing program, and information processing method
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2025-12-26
- Publication Date
- 2026-07-30
Smart Images
Figure JP2025045782_30072026_PF_FP_ABST
Abstract
Description
Information Processing Apparatus, Information Processing System, Information Processing Program, and Information Processing Method
[0001] The present disclosure relates to an information processing apparatus.
[0002] A cross-reality (XR) system that allows a user to experience virtual reality using a head-mounted display (HMD) has been proposed. Such an XR system is used for viewing videos, and in recent years, spatial videos that can be viewed as stereoscopic videos have become widespread. For example, Patent Document 1 discloses a technique for displaying a second virtual space on a first virtual space.
[0003] Japanese Patent Application Laid-Open No. 2023-148674
[0004] However, Patent Document 1 is a technique for displaying different virtual spaces, and it does not consider reaching out and interfering with those virtual spaces. For example, even if video content (image content) such as a spatial video is played in a window instead of the second virtual space and a hand is extended to interfere with an object shown in the video content, providing haptic feedback or the like is not considered. If it becomes possible to interfere with an object shown in video content, an improvement in the user experience is expected.
[0005] Therefore, the present disclosure has been made in view of these points, and an object thereof is to improve the user experience when interfering with video content.
[0006] One aspect of the present disclosure includes an acquisition unit that acquires the position of a user's hand, a control unit that controls an image content to be arranged in a virtual space, and a determination unit that determines an interference between the hand and a subject of the image content in the virtual space based on the position of the hand. When the hand first interferes with an edge of a predetermined plane related to the image content when starting to interfere with the image content, the determination unit does not determine that the hand has interfered with the subject even if the hand has interfered with the subject. The information processing apparatus is characterized by this.
[0007] According to the present disclosure, it is possible to improve the user experience when interfering with video content.
[0008] This is a diagram illustrating an information processing system according to the first embodiment of this disclosure. This is a diagram illustrating the internal configuration of an information processing device according to the first embodiment of this disclosure. This is a diagram illustrating the internal configuration of a user assistance device according to the first embodiment of this disclosure. This is a schematic diagram illustrating an example of a scene in which an image content is interfered with according to the first embodiment of this disclosure. This is a flowchart diagram illustrating the determination of interference with a virtual object according to the first embodiment of this disclosure. This is a schematic diagram illustrating an example of a scene in which a user's hand is touching a virtual object in a virtual space according to the first embodiment of this disclosure. This is a schematic diagram illustrating an example of a scene in which a user's hand is touching content in a spatial video in a virtual space according to the first embodiment of this disclosure. This is a diagram illustrating an example of a method for displaying a user's hand according to the first embodiment of this disclosure. This is a diagram illustrating an example of a method for displaying a user's hand according to the first embodiment of this disclosure. This is a diagram illustrating a specific example of determining a virtual object that a user's hand can interfere with according to the first embodiment of this disclosure. This is a diagram illustrating a specific example of determining a virtual object that a user's hand can interfere with according to the first embodiment of this disclosure.
[0009] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted from the drawings. The following embodiments are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications and various conditions (operating conditions, operating environment, etc.) of the device to which this disclosure applies.
[0010] In the following embodiments, identical components are denoted by the same reference numerals.
[0011] (First Embodiment) <Configuration of the Information Processing System> Referring to Figure 1, the information processing system according to Embodiment 1 will be described. The information processing system includes an HMD 100 and a user assistance device 110.
[0012] The HMD100 is a display device (electronic device) that can be worn on the user's head. The HMD100 is equipped with a camera for capturing images in front of the user and a display for showing images to the user. The HMD100 displays a composite image on the display, which is a fusion of the image captured by the camera and a virtual object that is computer graphics (CG) content. This allows the user to experience virtual reality with their own eyes. The HMD100 also has a function (hand tracking) that detects the user's hands from the captured images and acquires information about the position and posture of the hands, allowing the hand movements to interact with the virtual object. This allows the user to intuitively manipulate the virtual object using their hands.
[0013] In the first embodiment, an HMD is assumed as an example of a device constituting an information processing system, but this disclosure is not limited to these embodiments. For example, it may be a smartphone or tablet terminal equipped with a camera, or other devices such as a personal computer (PC) or digital camera.
[0014] The user assistance device 110 is a device for providing the user with tactile feedback instructed by the HMD 100. In the first embodiment, the user assistance device 110 is described as having a glove shape, but it may also be a bracelet shape, a ring shape, or the like.
[0015] <Internal Configuration of HMD100> The internal configuration of HMD100 will be described with reference to Figure 2A. HMD100 includes a control unit 200, a communication unit 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an imaging unit 204, and a display unit 205.
[0016] The control unit 200 controls each component of the HMD 100. The control unit 200 executes the program stored in the ROM 203, utilizes the RAM 202 as a work area, and controls the entire processing unit of the HMD 100. The control unit 200 has at least one CPU (Central Processing Unit) that executes the program stored in the ROM 203, and at least one other circuit. The control unit 200 may be composed of, for example, one or more processors such as CPUs or GPUs. Alternatively, instead of the control unit 200 controlling the entire device, multiple hardware components may share the processing to control the entire device.
[0017] The communication unit 201 is an interface for sending and receiving various data such as files and commands by communicating directly with the user assistance device 110 (communication unit 211) or external devices, or via the Internet. Wireless communication may be performed using wireless communication methods such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark), or communication may be performed via a wired connection.
[0018] RAM 202 is the main memory and temporary storage area of the HMD 100, including the work area. RAM 202 stores the spatial coordinates used for hand contact detection and the hand display method, which are determined based on the flowchart in Figure 4. RAM 202 consists of, for example, volatile memory (DRAM) using semiconductor elements.
[0019] ROM 203 is an electrically erasable and recordable non-volatile memory. When power is turned on to the HMD 100, the control unit 200 reads a program from ROM 203 and starts controlling the HMD 100. ROM 203 is, for example, a large-capacity NAND flash memory and stores control programs executed by the control unit 200 and spatial video data. Spatial video is image content that reproduces the depth relationship of the subject at the time of shooting. Although it is referred to as spatial video, it may also be still image content.
[0020] Although the control unit 200, RAM 202, and ROM 203 have been described here as separate hardware resources, these functions may also be integrated and implemented on a single LSI.
[0021] The imaging unit 204 includes two cameras (imaging devices). The two cameras are positioned near the user's left and right eyes when the HMD 100 is worn, in order to capture a space similar to the space the user normally sees. The monochrome or color images (captured images) captured by the two cameras of the subject (the area in front of the user) are output to the RAM 202. The control unit 200 also acquires distance information to the subject by measuring distance with the stereo camera. The imaging unit 204 is composed of, for example, an optical lens unit, an optical system that controls aperture, zoom, focus, etc., and an image sensor that converts the light (image) introduced through the optical lens unit into an electrical image signal. Generally, a CMOS image sensor (CMOS image sensor) using CMOS or a CCD image sensor (CCD image sensor) using CCD is used as the image sensor.
[0022] The display unit 205 displays a composite image of the captured image and virtual objects, as well as operation menus for controlling the HMD 100, as a three-dimensional image. The display unit 205 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, an organic EL panel is positioned in front of each of the user's eyes. It is also possible to use a device with a semi-transparent half-mirror for the display unit 205. In this case, for example, the display unit 205 may display an image so that virtual objects are superimposed directly onto the real space visible through the half-mirror, using a technology generally known as AR (Augmented Reality). Alternatively, the display unit 205 may display an image of a completely virtual space without using captured images, using a technology generally known as VR (Virtual Reality).
[0023] The attitude sensor unit 206 acquires attitude (and position) information of the HMD 100. The attitude sensor unit 206 then acquires attitude information of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100. The attitude sensor unit 206 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 206 is used when acquiring user attitude information (attitude information), and the control unit 200 controls it to acquire user attitude information (attitude information).
[0024] Here, the HMD100 is described as an example of a head-mounted display device with a built-in information processing device, but the information processing device is not limited to this. The information processing device connected to the head-mounted display device by wire or wireless means may be a smartphone, tablet terminal, or PC. In this case, some of the processing performed by the HMD100 described above may be performed by the information processing device connected to the head-mounted display device by wire or wireless means.
[0025] <Internal Configuration of User Assistance Device 110> The internal configuration of the user assistance device 110 will be explained with reference to Figure 2B.
[0026] The user assistance device 110 includes a control unit 210, a communication unit 211, and a haptic feedback unit 212.
[0027] The control unit 210 is a CPU that controls each component of the user assistance device 110. The control unit 210 controls the haptic feedback display unit 212 according to the haptic feedback parameters obtained from the HMD 100 via the communication unit 211. Alternatively, instead of the control unit 210 controlling the entire device, multiple hardware components may share the processing to control the entire device. Alternatively, instead of the control unit 210 controlling the user assistance device 110, the control unit 200 of the HMD 100 may control the user assistance device 110.
[0028] The communication unit 211 communicates with the HMD 100 (communication unit 201).
[0029] The haptic feedback unit 212 has a motor that provides haptic feedback information and, according to the provided haptic feedback parameters, provides the user with a desired feel or sensation of force, object, weight, movement, or shape change. The haptic feedback parameters are not limited to the force of mass, but may also be based on information regarding the physical characteristics of the virtual object. The physical characteristics may include at least one of the following: size, shape, stiffness, weight, thermal characteristics, or texture of the virtual object.
[0030] The attitude sensor unit 213 acquires attitude (and position) information of the user assistance device 110. The attitude sensor unit 213 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The control unit 210 acquires the attitude (and position) information of the user assistance device 110 and controls it to transmit it to the control unit 200 via the communication unit 211. Note that the user assistance device 110 does not necessarily have an attitude sensor unit 213; in that case, the control unit 200 of the HMD 100 acquires (estimates) the attitude (and position) information of the user assistance device 110 based on the captured image taken by the imaging unit 204.
[0031] <Examples of HMD and User Assistive Device Use> Referring to Figure 3, an example of the use of the information processing system in Embodiment 1 of this disclosure will be described. Figure 3 is a view of the user from directly above. In Figure 3, it is assumed that the user is wearing the HMD 100 and user assistive device 110 and experiencing virtual reality (virtual space).
[0032] The virtual space contains virtual objects and operation menus, and their three-dimensional position coordinates are included in the spatial coordinates 300. The spatial coordinates 300 are used, for example, to interact with the virtual objects and operation menus through hand tracking.
[0033] The user's field of view 301 is the range of the view visible to the user and the range of the image displayed on the display unit 205.
[0034] The spatial video window 302 is a virtual object for viewing spatial video. In Figure 3, the spatial video window 302 is positioned within the user's field of view 301. The spatial video is displayed in this spatial video window 302, allowing the user to view the image.
[0035] The spatial coordinates 303 of the spatial video represent the three-dimensional positional coordinates of the feature points of the subjects included in the spatial video's image. The control unit 200 acquires these coordinates based on distance information obtained by measuring the distance of the stereo camera image of the spatial video.
[0036] The user viewing range 304 of the spatial video is the range of the spatial video image that is visible to the user. The visible range differs depending on the user's viewing position, and the range of the image displayed in the spatial video window 302 changes as the user moves.
[0037] <HMD Processing Procedure> The processing procedure of the information processing system in Embodiment 1 of this disclosure will be explained with reference to the flowchart in Figure 4. This flowchart starts when a virtual object is displayed in the virtual space and is executed at least the first time. Note that the timing of execution of this flowchart is not limited to the timing when a spatial video window is displayed in the virtual space. For example, it may be the timing when the user starts the HMD or when the user starts an application for the HMD. Note that the following flowchart may be executed in response to a reset operation performed by the user at a predetermined timing. Note that the following flowchart may be executed not only before the spatial video window is placed, but also after the spatial video window is placed.
[0038] In step S401, the control unit 200 starts displaying the virtual space on the display unit 205 in response to the user's operation to start using the information processing system. The control unit 200 switches the spatial coordinates used for contact detection (interference detection) under predetermined conditions. Initially, the spatial coordinates used for contact detection are stored in the virtual space spatial coordinates 300 and RAM 202. The control unit 200 also switches the display method of the user's hand under predetermined conditions. For example, it is possible to switch between a format that displays the entire hand and a format that makes the hand semi-transparent, but other display methods may also be used. Immediately after startup, the hand is stored in the format that displays the entire hand.
[0039] In step S402, the control unit 200 controls the imaging unit 204 to capture an image of the user's surroundings. The control unit 200 detects the user's hand from the captured image and acquires information regarding the position and orientation of the user's hand in the virtual space.
[0040] In step S403, the control unit 200 obtains the position of the spatial video window 302 in the virtual space. If the spatial video window 302 is not placed in the virtual space, the control unit 200 places the spatial video window 302 and obtains its position.
[0041] In step S404, the control unit 200 determines whether the user's hand has come into contact with a virtual object using the spatial coordinates stored in the RAM 202. That is, the control unit 200 performs contact detection between the user's hand and a virtual object. Examples of virtual objects include the spatial video window 302, subjects included in the video displayed in the spatial video window 302, and virtual objects other than the spatial video window 302. In this flow, when contact detection is performed using the spatial coordinates of the spatial video, contact detection is performed with subjects shown in the spatial video, and contact detection is not performed with virtual objects other than virtual objects related to the spatial video. Also, when contact detection is performed using the spatial coordinates of the virtual space, contact detection is performed with virtual objects other than virtual objects related to the spatial video, and contact detection is not performed with subjects shown in the spatial video. That is, while contact detection is performed using the spatial coordinates of the spatial video, even if the user touches (interferes with) a virtual object such as an interior piece placed in the virtual space, it will not be determined as contact. Similarly, while contact detection is performed using the spatial coordinates of the virtual space, even if the user touches (interferes with) a subject shown in the spatial video, it will not be determined as contact.
[0042] Referring to Figure 5A, we will now explain a scenario in which contact detection is performed using spatial coordinates 300 in the virtual space, and the user's hand interferes with a virtual object in the virtual space. In Figure 5A, there are virtual objects 500 other than the spatial video window 302 in the virtual space, and they are included in the field of view 301 of the HMD 100. Also, if the spatial video window 302 is an opaque screen, the field of view 304 is the range in which the area behind the spatial video window 302 cannot be seen. The spatial coordinates 300 also include coordinates based on the virtual object 500. When the user reaches out to touch the virtual object 500, the position of the user's hand approaches the spatial coordinates that represent the virtual object 500, and interference (contact) occurs. In step S404, if the control unit 200 detects interference, it may send a control signal to the user assistance device 110 to generate haptic feedback corresponding to the virtual object 500 and provide it to the user.
[0043] Next, referring to Figure 5B, a scenario in which a user's hand interferes with content contained in a spatial video by performing contact detection using the spatial coordinates 303 of the spatial video will be explained. The spatial video content 501 is the video content contained in the spatial video image. The spatial coordinates 303 of the spatial video also include coordinates based on the spatial video content 501. When a user extends their hand into the spatial video window 302 and reaches out to touch the spatial video content 501, the position of the user's hand approaches the spatial coordinates that represent the spatial video content 501, causing interference. When the control unit 200 detects interference, it sends a control signal to the user assistance device 110, which generates haptic feedback corresponding to the spatial video content 501 and provides it to the user. Contact detection is performed even if the spatial video content 501 is located outside the user's spatial video field of view 304. Furthermore, a method is disclosed for predicting the position of the user's hand a few seconds later from its movement. This method may be used to sense the future position of the user's hand and prepare in advance to switch the spatial coordinates used for contact detection.
[0044] In step S405, the control unit 200 performs the display of the hand according to the hand display method stored in the RAM 202. Here, FIGS. 6A and 6B show an example of the hand display method when the user extends a hand into the spatial video window. FIG. 6A is an example when the spatial video window 302 is viewed from slightly to the right of the front. The user's hand 600 is the user's hand in the virtual space and extends into the spatial video window 302. The hand 601 is the part extending into the spatial video window and is semi-transparently displayed. In the case of an AR space or an MR (Mixed Reality) space, a method of superimposing computer graphics on the video of the actual user's hand and performing blurring or mosaic processing can be considered. Additionally, a method of changing the color of the hand to, for example, black and white, obtaining the background video behind the hand from another camera video, or creating a pseudo-background video behind the hand from the video around the hand and superimposing and displaying it to make it semi-transparent (processing to increase the transparency) or invisible can be considered. In the case of a VR space, methods of performing blurring or mosaic processing on the CG video of the avatar corresponding to the user's hand, changing the CG video of the hand to semi-transparent or another color, and making the CG video of the hand invisible can be considered. Also, FIG. 6B is an example when the spatial video window 302 is viewed from the front. In this case, although the hand 600 extends into the spatial video window 302, the hand does not penetrate into the depth of the spatial video window 302, so the hand is displayed as it is.
[0045] In particular, consider the case of the hand display method described in step S408 below (when switched to hidden or semi-transparent). As shown in Figure 6A, when the user is reaching into the spatial video and the back of the spatial video is visible, the hand will be visible behind the spatial video in the MR space. Originally, the spatial video's image space extends beyond the spatial video window, so when the user is reaching into the spatial video, it may appear as if the hand is sticking out of the spatial video window, which can impair the user experience. Therefore, when the user is reaching into the spatial video and the back of the spatial video is visible, computer graphics are overlaid on the image of the user's real hand, and blurring or mosaic processing is applied to hide the hand sticking out of the spatial video window. This prevents the hand from appearing to stick out behind the spatial video window, improving the user experience. Note that, as shown in Figure 6B, when the user is viewing the spatial video from the front and the hand is contained within the spatial video, the hand is displayed without blurring or mosaic processing to hide it. Additionally, parts of the user's hands and arms that appear to be within the spatial video will be displayed as normal, while parts that extend beyond the spatial video will be blurred or pixelated to conceal the hands.
[0046] In step S406, the control unit 200 determines whether the user's hand moved into the spatial video area by passing through the frame (the edge of the window surface) without touching the outer frame (the edge of the window surface) of the spatial video window. If the control unit 200 determines that the user's hand moved into the spatial video area by passing through the frame without touching the outer frame of the spatial video window, the process proceeds to step S407. If the control unit 200 does not determine that the user's hand moved into the spatial video area by passing through the frame without touching the outer frame of the spatial video window, the process proceeds to step S409. In this embodiment, it is expressed as the outer frame of the spatial video window, but it may also be the edge of a predetermined surface. That is, it is not limited to the outer frame of the spatial video; a frame (edge of a surface) for making the above determination may be provided. Furthermore, such a frame (edge of a surface) may or may not be visible to the user.
[0047] In step S407, when the control unit 200 determines that the user's hand has passed through in the manner of step S406, the control unit 200 switches the spatial coordinates used for contact determination stored in the RAM 202 to the spatial coordinates 303 of the spatial video. That is, the control unit 200 determines interference with the subject shown in the spatial video and does not determine interference with a virtual object different from the virtual object related to the spatial video.
[0048] In step S408, when the control unit 200 determines that the user's hand has passed through in the manner of S406, the control unit 200 switches the display method of the user's hand stored in the RAM 202 to blurring, mosaic processing, semi-transparency, or non-display.
[0049] In step S409, the control unit 200 determines whether the user's hand has moved out of the spatial video window 302 regardless of whether the user's hand has touched the outer frame of the spatial video window. If there is no interference between the user's hand and the spatial video window 302, the user's hand is determined to be outside the frame. When the control unit 200 determines that the user's hand has moved out of the spatial video window 302 regardless of whether the user's hand has touched the outer frame of the spatial video window, the process proceeds to step S410. When the control unit 200 does not determine that the user's hand has moved out of the spatial video window 302 regardless of whether the user's hand has touched the outer frame of the spatial video window, the process proceeds to step S412.
[0050] In step S410, when the control unit 200 determines that the user's hand is outside the frame by the method of step S409, the control unit 200 switches the spatial coordinates used for contact determination stored in the RAM 202 to the spatial coordinates 300 in the virtual space. That is, the control unit 200 determines interference with a virtual object different from the virtual object related to the spatial video and does not determine interference with the subject shown in the spatial video.
[0051] In step S411, if the control unit 200 determines that the user's hand is outside the frame by the method in step S409, it switches the display method of the user's hand stored in RAM 202 to a normal display without blurring or mosaic effects.
[0052] In step S412, the control unit 200 determines whether or not to terminate the system. If the control unit 200 determines to terminate the system, it terminates this flow; otherwise, it proceeds to step S402.
[0053] The HMD processing procedure has been explained above with reference to the flowchart in Figure 4. According to the above flow, contact with a subject in the spatial video is determined only when the user's hand first interferes with the display surface of the spatial video window and moves into the spatial video area by passing through the frame without touching the spatial video window frame. Also, if the hand is extended into the spatial video but appears to be penetrating into the background of the spatial video, the display is changed to hide the part of the hand that is penetrating into the background of the spatial video. Furthermore, if the user first touches the spatial video window frame before moving into the spatial video area, even if they interfere with a subject in the spatial video, it will not be determined that they have interfered with the subject. Here, the timing of first touching the spatial video window frame refers to the first timing in the time interval in which the user is interfering with the virtual object related to the spatial video. In other words, the timing when the user stops interfering with the spatial video window after being in a state of interfering with the spatial video window is the timing of the end of the time interval in which the user is interfering with the virtual object related to the spatial video. After the HMD is started, the user will interfere with the spatial video window, stop interfering with the spatial video window, and then start interfering with the spatial video window again, at which point the timing of first touching the spatial video window frame will occur again.
[0054] The above-described flow describes a flow in which, in addition to providing haptic feedback when contact is detected, the method of displaying the hand is also switched. However, this is not the only option. The method of displaying the hand may not be changed, and haptic feedback may be provided instead. In other words, steps S405, S408, and S411 may be omitted. Alternatively, the method of displaying the hand may be switched without providing haptic feedback. In other words, steps S404, S407, and S410 may be omitted.
[0055] In the above-described flow, the process is switched based on whether or not the spatial video window frame is touched, but this is not the only option. For example, it is not limited to spatial video; it could also be planar video content, planar image content, or a three-dimensionally displayed image. Alternatively, instead of the spatial video window frame, a predetermined surface used for the above determination could be defined, and the process could be switched based on whether or not the edge of the predetermined surface is touched first.
[0056] <Specific example of determining virtual objects that can be interfered with by a hand> In Figure 6A above, the user extends their hand to the center of the spatial video window 302 without touching the outer frame, and the hand passes through to the other side of the spatial video window 302. In this state, interference with a subject inside the spatial video window is determined. The part of the hand that has passed through to the other side of the spatial video window 302 is considered to be in a passing state, and processing such as transparency (processing to increase transparency), blurring, and mosaic is applied to the part of the hand that has passed through, as described above.
[0057] The following describes specific examples of how to determine virtual objects that a hand can interact with, with reference to Figures 7A, 7B, and 7C. Specifically, depending on the type of image content, we will explain cases where interaction is possible with virtual objects related to the image content, and cases where interaction is possible with virtual objects different from those related to the image content.
[0058] Figure 7A illustrates how the user's hand is displayed when it first touches the outer frame of the spatial video window 302 in the user's field of view 301. When the user's hand first touches the outer frame of the spatial video window 302, even if the hand passes to the other side of the spatial video window 302, the hand is not considered to have passed through. That is, the hand is displayed without applying processing such as transparency, blurring, and mosaic. Furthermore, various methods can be considered for determining whether the hand has passed through, and examples of other implementation methods are shown in Figures 7B and 7C.
[0059] Figure 7B assumes a scenario in which a spatial video window 702 and a subject 701 floating in front of the spatial video are placed in the virtual space within the user's field of view 301. The subject 701 is smaller in shape than the spatial video window 702 and is placed adjacent to it. In this case, if the user's hand first touches the spatial video window 702 and then touches the subject 701, it is determined that the user has touched the subject 701. That is, if the user's hand first touches the spatial video window 702 and then touches the subject 701, the hand passing through the other side of the spatial video window 702 is considered to be in a passing state.
[0060] Figure 7C assumes a scenario in which a spatial video window 703 and a subject 704 positioned behind the spatial video are placed in the virtual space within the user's field of view 301. The spatial video window 703 is positioned in front of the subject 704 and has a donut-shaped hollow frame, for example, a square shape. In this case, if the user's hand first touches the spatial video window 703 and then touches the subject 704, it is determined that the hand has touched the subject 704. That is, if the user's hand first touches the spatial video window 703 and then touches the subject 704, the hand passing beyond the spatial video window 703 or the subject 704 is considered to be in a passing state.
[0061] (Modification) In the first embodiment described above, an example was explained in which, if the user's hand first interferes with the frame of the spatial video window, even if it subsequently interferes with a subject in the spatial video, it is not determined that the subject has been interfered with. However, the system may also be configured so that, even if the user's hand first interferes with the right or left side of the frame of the spatial video window, it is not determined that the subject has been interfered with. In other words, if the user's hand first interferes with the upper or lower side of the frame of the spatial video window, it may be determined that the subject has been interfered with if it interferes with a subject in the spatial video. When the spatial video is placed close to the user, when the user raises their hand from a lowered position to begin interfering with the spatial video, they may first touch the lower side of the frame of the spatial video even though they intend to interfere with the spatial video. In such cases, even if the subject is interfered with, it will not be determined that the subject has been interfered with, so the user may fold their hand and raise it, or move a predetermined distance away from the spatial video before interfering with it, which may impair the user experience. Therefore, as described above, if the user's hand first interferes with the lower side of the frame of the spatial video window, it may be determined that the hand has interfered with the subject shown in the spatial video. Furthermore, if the distance between the user (HMD) and the spatial video window is less than a threshold, interference with the subject shown in the spatial video may be determined even if the user's hand first interferes with the upper or lower side of the frame of the spatial video window. Also, if the distance between the user (HMD) and the spatial video window is not less than a threshold, if the user's hand first interferes with the frame of the spatial video window, it may be determined that the hand has not interfered with the subject shown in the spatial video, even if it subsequently interferes with the subject. In other words, the frame used as a trigger to prevent the detection of hand interference with a subject does not have to consist of four sides like a rectangle, but may consist of at least one side.
[0062] Furthermore, the first embodiment described above does not explain how to display the spatial video when the user's hand first interferes with the frame of the spatial video window, nor does it specifically mention changing the display method. However, the display method of the spatial video may be changed when the user's hand first interferes with the frame of the spatial video window, i.e., as shown in Figure 7A. For example, if the user's hand first interferes with the frame of the spatial video window, even if it interferes with the subject of the spatial video, it will not be determined that it has interfered with the subject, so the transparency of the spatial video may be increased. Alternatively, instead of increasing the transparency, the spatial video may be hidden. In this way, the user can visually recognize that they cannot interfere with the spatial video, and if they want to interfere with the spatial video, they can try again. Also, if the user's hand first interferes with the inside of the display surface of the spatial video window, the transparency of the spatial video will be maintained and displayed without increasing its transparency. Note that the frame of the spatial video and the display surface are just examples, and a predetermined surface used for the above determination may be defined, and the processing may be switched depending on whether the user first touches the edge of the predetermined surface or the inside of the predetermined surface. The predetermined surface is a virtual display that displays the image content, and the edges of the predetermined surface may be the edges of the image area of the virtual display.
[0063] (Other Embodiments) The present disclosure can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present disclosure.
[0064] While the present disclosure has been described in detail above based on its preferred embodiments, the present disclosure is not limited to these specific embodiments, and various forms that do not depart from the gist of the disclosure are also included. Some of the embodiments described above may be combined as appropriate.
[0065] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of a single functional unit may be realized by individual hardware. Two or more functions of a single functional unit may be realized by common hardware. In addition, each functional unit may or may not be realized by hardware such as ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.
[0066] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.
[0067] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).
[0068] This embodiment includes the following configurations, methods, and programs.
[0069] [Configuration 1] An information processing device comprising: an acquisition means for acquiring the position of a user's hand; a control means for controlling the placement of image content in a virtual space; and a determination means for determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein the determination means does not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject.
[0070] [Configuration 2] The information processing apparatus according to Configuration 1, characterized in that the determination means does not determine whether the hand and the subject have interfered if the hand first interferes with the edge when it begins to interfere with the image content.
[0071] [Configuration 3] The information processing apparatus according to Configuration 1 or 2, characterized in that the determination means determines whether or not the hand has interfered with the subject if the hand does not first interfere with the edge of the image content when it begins to interfere with the image content.
[0072] [Configuration 4] The information processing apparatus according to any one of Configurations 1 to 3, characterized in that the determination means determines interference between the hand and the subject if the hand first interferes with the upper or lower side of the edge when the hand begins to interfere with the image content, and does not determine that the hand has interfered with the subject even if the hand interferes with the subject if the hand first interferes with the right or left side of the edge when the hand begins to interfere with the image content.
[0073] [Configuration 5] The information processing apparatus according to any one of Configurations 1 to 4, characterized in that when the distance between the user and the image content is less than a threshold, the determination means determines that there is interference between the hand and the subject if the hand first interferes with the upper or lower side of the edge when it begins to interfere with the image content, and does not determine that the hand has interfered with the subject even if the hand interferes with the subject if the hand first interferes with the right or left side of the edge when it begins to interfere with the image content.
[0074] [Configuration 6] The information processing device according to any one of Configurations 1 to 5, further comprising a generation means for generating haptic feedback when the determination means determines that the hand and the subject have interfered with each other.
[0075] [Configuration 7] The information processing device according to any one of Configurations 1 to 6, characterized in that the display of the subject is changed when the determination means determines that the hand and the subject have interfered with each other.
[0076] [Configuration 8] The information processing apparatus according to any one of Configurations 1 to 7, characterized in that the subject is a subject located behind the image content.
[0077] [Configuration 9] The information processing apparatus according to any one of Configurations 1 to 8, characterized in that the acquisition means acquires the position of the hand based on the captured image.
[0078] [Configuration 10] The information processing apparatus according to any one of Configurations 1 to 9, characterized in that the image content is image content that reproduces the depth relationship of the subject at the time of shooting.
[0079] [Configuration 11] The information processing apparatus according to any one of Configurations 1 to 10, characterized in that the control means controls the transparency of the image content or hides the image content when the hand first interferes with the image content, specifically when the hand begins to interfere with the image content and first interferes with the edge of a predetermined surface relating to the image content.
[0080] [Configuration 12] The information processing apparatus according to Configuration 11, characterized in that the control means maintains the transparency of the image content when the hand first interferes with the image content on the inside of a predetermined surface relating to the image content.
[0081] [Configuration 13] The information processing apparatus according to any one of Configurations 1 to 12, characterized in that the predetermined surface is a virtual display for displaying the image content, and the edge is the edge of the image area of the virtual display.
[0082] [Configuration 14] An information processing device comprising: an acquisition means for acquiring the position of a user's hand; a control means for controlling the placement of image content in a virtual space; and a determination means for determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein the determination means determines that the hand has interfered with the subject if the hand interferes with the inside of a predetermined surface relating to the image content before interfering with the subject.
[0083] [Method] An information processing device comprising: an acquisition step of acquiring the position of a user's hand; a control step of controlling the placement of image content in a virtual space; and a determination step of determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein in the determination step, if the hand first interferes with the edge of a predetermined surface relating to the image content when it begins to interfere with the image content, the device does not determine that the hand has interfered with the subject, even if the hand then interferes with the subject.
[0084] [Program] A program for causing a computer to function as one of the means of the information processing device described in any one of items 1 to 14.
[0085] [System] An information processing system comprising: an acquisition device for acquiring the position of a user's hand; a control device for controlling the placement of image content in a virtual space; and a determination device for determining interference between the hand and the subject of the image content in the virtual space based on the position of the hand, wherein the determination device does not determine that the hand has interfered with the subject if, when the hand begins to interfere with the image content, it first interferes with the edge of a predetermined surface relating to the image content, even if the hand then interferes with the subject.
[0086] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are attached to make the scope of the invention public.
[0087] This application claims priority based on Japanese Patent Application No. 2025-009434, filed on 22 January 2025, and all of its contents are incorporated herein by reference.
Claims
1. An information processing device comprising: acquisition means for acquiring the position of a user's hand; control means for controlling the placement of image content in a virtual space; and generation means for generating a tactile effect when the hand comes into contact with the subject of the image content in the virtual space based on the position of the hand, wherein the generation means does not generate the tactile effect even if the hand passes over the edge of a predetermined surface relating to the image content and comes into contact with the subject.
2. The information processing apparatus according to claim 1, further comprising determination means for determining whether the hand and the subject have come into contact based on the position of the hand, wherein the determination means does not determine whether the hand and the subject have come into contact if the hand first comes into contact with the edge when the hand comes into contact with the image content.
3. The information processing apparatus according to claim 1, further comprising determination means for determining whether the hand and the subject have come into contact based on the position of the hand, wherein the determination means determines whether the hand has come into contact with the subject if the hand does not first come into contact with the edge of the image content when it comes into contact with the image content.
4. The information processing apparatus according to claim 1, characterized in that the generating means generates the tactile effect if the hand first touches the upper or lower edge when it touches the image content, and does not generate the tactile effect even if the hand touches the subject if the hand first touches the right or left edge when it touches the image content.
5. The information processing apparatus according to claim 1, characterized in that the generating means generates the tactile effect when the distance between the user and the image content is less than a threshold, and when the hand touches the image content, if the hand first touches the upper or lower edge, it does not generate the tactile effect even if the hand touches the subject when the hand first touches the right or left edge of the image content.
6. The information processing apparatus according to claim 1, further comprising display control means for changing the display of the subject when the hand comes into contact with the subject.
7. The information processing apparatus according to claim 1, characterized in that the subject is a subject located behind the image content.
8. The information processing apparatus according to claim 1, characterized in that the acquisition means acquires the position of the hand based on the captured image.
9. The information processing apparatus according to claim 1, characterized in that the image content is image content that reproduces the depth relationship of the subject at the time of shooting.
10. The information processing apparatus according to claim 1, characterized in that the control means controls the transparency of the image content to be increased or the image content to be hidden when the hand first makes contact with the image content and the hand first makes contact with the edge of a predetermined surface relating to the image content.
11. The information processing apparatus according to claim 10, characterized in that the control means maintains the transparency of the image content when the hand first makes contact with the image content, on the inside of a predetermined surface relating to the image content.
12. The information processing apparatus according to claim 1, characterized in that the predetermined surface is a virtual display for displaying the image content, and the edge is the edge of the image area of the virtual display.
13. An information processing apparatus comprising: acquisition means for acquiring the position of a user's hand; control means for controlling the placement of image content in a virtual space; and generation means for generating a tactile effect when the hand in the virtual space touches the subject of the image content based on the position of the hand, wherein the generation means determines and generates the tactile effect when the hand passes inside a predetermined surface relating to the image content and touches the subject.
14. An information processing apparatus comprising: acquisition means for acquiring the position of a user's hand; control means for controlling the placement of image content in a virtual space; and display control means for changing the display of a subject in the image content when the hand comes into contact with the subject in the virtual space based on the position of the hand, wherein the display control means does not change the display of the subject when the hand passes inside a predetermined surface relating to the image content and comes into contact with the subject.
15. An information processing method comprising: an acquisition step of obtaining the position of a user's hand; a control step of controlling the placement of image content in a virtual space; and a generation step of generating a tactile effect when the hand in the virtual space touches the subject of the image content based on the position of the hand, wherein the generation step does not generate the tactile effect even if the hand passes over the edge of a predetermined surface relating to the image content and touches the subject.
16. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.
17. An information processing system comprising: an acquisition device for acquiring the position of a user's hand; a control device for controlling the placement of image content in a virtual space; and a generation device for generating a tactile effect when the hand in the virtual space touches the subject of the image content based on the position of the hand, wherein the generation device does not generate the tactile effect even if the hand passes over the edge of a predetermined surface relating to the image content and touches the subject.