Tactile presentation system, tactile presentation method, and program

Abstract

The present invention allows the position of a virtual object to be intuitively perceived.　This tactile presentation system determines, for each of at least four vibration devices worn at a distance from each other by a user, the strength of vibration on the basis of information indicating the main direction in charge of the vibration device, the direction of a virtual object with respect to the vibration device, and information indicating the distance between the virtual object and the vibration device in a virtual space, and independently vibrates the at least four vibration devices in accordance with the determined strength of vibration.

Description

Haptic presentation system, haptic presentation method and program

[0001] The present disclosure relates to a haptic presentation system, a haptic presentation method, and a program.

[0002] There is a technique of causing a user to sense a direction or provide a user with a sensation of a specific motion by vibrating a plurality of vibration motors provided on a user's head.

[0003] Non-Patent Document 1 discloses attaching a plurality of vibrators to a human head and presenting directions to the person via haptic stimulation. Non-Patent Document 2 discloses arranging a plurality of vibration motors around the head, and allowing a user to sense teleportation in a virtual space and wind pressure from a ball through haptic feedback.

[0004] Kazuhiro Matsuoka, Masayuki Tsuruta, Hirotaka Ishii, Hiroshi Shimoda, Hidekazu Yoshikawa, "Experimental Study on Direction Information Presentation Method Using Vibration Stimulus to the Head", Human Interface Symposium 2006, Vol.1, No.1312, p139-p144, 2006Shao-Yu Chu et al., "MotionRing: Creating Illusory Tactile Motion around the Head using 360° Vibrotactile Headbands", UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology, Pages 724-731, 2021

[0005] The inventors have studied notifying a user who uses a virtual space of the presence of a virtual object existing around the user using vibration. However, with existing methods, it has been difficult to allow the user to intuitively perceive the position of the virtual object.

[0006] One object of the present disclosure is to provide a technique that enables intuitive perception of the position of a virtual object.

[0007] (1) A tactile presentation system comprising: vibration determination means for determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device; and control means for independently vibrating the at least four vibration devices according to the determined intensity of vibration.

[0008] (2) In (1), the vibration determination means determines the start time of the vibration according to the start time of playback of the sound associated with the virtual object, a tactile presentation system.

[0009] (3) A tactile presentation system in which the vibration determination means determines the vibration pattern of each of the at least four vibration devices based on the virtual object, and the control means causes each of the at least four vibration devices to reproduce the determined vibration pattern.

[0010] (4) A tactile presentation system further comprising, in (3), a generation means for generating vibration patterns for each of the at least four vibration devices based on the sound associated with the virtual object.

[0011] (5)(3) or (4), the control means outputs sound associated with the virtual object to headphones in synchronization with the playback of the vibration pattern, a haptic presentation system.

[0012] (6) A haptic presentation method comprising: determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device; and independently vibrating the at least four vibration devices according to the determined intensity of vibration.

[0013] (7) A program for causing a computer to function as a vibration determination means for determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device, and a control means for causing the at least four vibration devices to vibrate independently according to the determined vibration intensity.

[0014] This figure shows an example of the configuration of the haptic presentation system according to this embodiment. This figure illustrates the configuration of the haptic device. This block diagram shows the functions realized by the haptic presentation system. This flowchart shows an overview of the processing of the control device. This flowchart explains the processing related to the output of sound and vibration. This figure shows an example of the process for determining the magnitude of vibration. This figure illustrates an example of a method for calculating the magnitude of vibration. This figure illustrates another example of a method for calculating the magnitude of vibration.

[0015] Embodiments of the present invention will be described below with reference to the drawings. Components having the same function will be denoted by the same reference numeral, and their descriptions will be omitted.

[0016] Figure 1 shows an example of the configuration of the haptic feedback system according to this embodiment. The haptic feedback system according to the present invention includes a control device 10, a display device 20, a haptic device 30, and an audio output device 40. The haptic feedback system draws a three-dimensional image based on the user's position and orientation in a virtual space, and information about other objects, and displays the image on the display device 20. The haptic device 30 also presents tactile sensations to the user according to the state of other objects in the virtual space. Hereinafter, other objects in the virtual space will be referred to as "virtual objects." The audio output device 40 outputs sound to the user according to the state of other objects in the virtual space. Hereinafter, the display device 20 and the audio output device 40 are not included in the haptic feedback system and may be connected from outside the haptic feedback system.

[0017] The control device 10 controls the display on the display device 20 and also controls the output of the haptic device 30 and the audio output device 40. The control device 10 is a computer, such as a personal computer or a home game console. The control device 10 includes a processor 11, storage 12, a communication unit 13, and an input / output unit 14.

[0018] The processor 11 operates according to a program stored in the storage 12 and controls the communication unit 13, the input / output unit 14, and the display device 20. The program may be provided by another computer via communication through the communication unit 13, or it may be provided to another computer by being stored in a computer-readable storage medium such as flash memory or an optical disc.

[0019] The storage 12 is composed of at least a portion of memory (e.g., DRAM, non-volatile memory) and a secondary storage device (e.g., a hard disk drive, a solid-state drive). The storage 12 stores the above-mentioned program. The storage 12 also stores information and calculation results input from the processor 11, the communication unit 13, etc.

[0020] The communication unit 13 consists of integrated circuits, terminals, antennas, etc., for communicating with other devices. The communication unit 13 is configured to communicate with other devices (e.g., computers) according to protocols such as Ethernet, Wi-Fi®, or Bluetooth®. The communication unit 13 may include, for example, a network interface card. Based on the control of the processor 11, the communication unit 13 inputs information received from other devices to the processor 11 and storage 12, and transmits the information to the other devices.

[0021] The input / output unit 14 acquires information from or transmits information to an external device. The input / output unit 14 controls an external device. The input / output unit 14 may include, for example, a USB (Universal Serial Bus) integrated circuit and terminals. At least some of the following may be connected to the input / output unit 14: an input device such as a keyboard or game controller, a haptic device 30, and an audio output device 40.

[0022] The display device 20 is a device that displays images to the user. The display device 20 may be a flat panel display, a head-mounted display, or it may be built into the control device 10.

[0023] The tactile device 30 is a device that presents tactile sensations around the head through vibration. The tactile device 30 includes a vibration control circuit 31 and a plurality of vibration devices 32. The vibration control circuit 31 controls the operation of the plurality of vibration devices 32. The vibration control circuit 31 may be, for example, a D / A converter. The vibration control circuit 31 may be located outside the tactile device 30 or may be built into the control device 10.

[0024] Figure 2 is a diagram illustrating the configuration of the tactile device 30. Figure 2 is a top view of the tactile device 30. The tactile device 30 further includes a band 33 to which multiple vibration devices 32 are fixed. The tactile device 30 is fixed to the head by the band 33, and the multiple vibration devices 32 are spaced apart from each other when worn by the user. The multiple vibration devices 32 are connected to a vibration control circuit 31 by wiring.

[0025] The band 33 is annular, and the vibration devices 32 are arranged such that, when viewed from the central axis C of the head, the direction in which adjacent vibration devices 32 are positioned differs by a predetermined angle (for example, at equal intervals). The central axis C of the head is an axis extending vertically from the center of the head. Each of the vibration devices 32 may be, for example, a voice motor coil. In the example in Figure 2, there are 8 vibration devices 32, but there can be 4 or more. The angle between adjacent vibration devices 32 when viewed from the annular center (corresponding to the central axis C) differs by an angle obtained by dividing 360° by the number of vibration devices 32 (45° in Figure 2). Of course, there may be some variation in that angle.

[0026] The audio output device 40 is a device that outputs sound, such as headphones, earphones, or speakers. The audio output device 40 includes an audio control circuit 41 and a driver unit (not shown). The driver unit outputs sound by driving an internal diaphragm. The audio control circuit 41 controls the driver unit. The audio output device 40 may output sound independently to the left and right ears.

[0027] Next, an overview of the processing of the haptic feedback system will be described. Figure 3 is a block diagram showing the functions realized by the haptic feedback system. Functionally, the haptic feedback system includes an object control unit 51, a vibration sound determination unit 52, a vibration sound control unit 53, a vibration conversion unit 55, a sound database 61, and a vibration database 62. The object control unit 51, vibration sound determination unit 52, vibration sound control unit 53, and vibration conversion unit 55 are mainly realized by the processor 11 of the control device 10 executing programs corresponding to each function stored in the storage 12 and controlling the communication unit 13, etc. The sound database 61 and vibration database 62 are stored in the storage 12.

[0028] The object control unit 51 determines the positions of the user and objects other than the user in the virtual space. The object control unit 51 also determines the orientation of the user and objects as needed. The object control unit 51 may include a so-called physics engine. The object control unit 51 may determine the positions of the user and objects based on so-called motion data, or it may determine the position of the user or objects based on input from the user via an input device, relative position to the user, or physical laws.

[0029] The vibration sound determination unit 52 determines the vibration intensity for each of the multiple vibration devices 32 based on information indicating the primary direction of the vibration device 32 in the virtual space, the direction of the virtual object relative to the vibration device 32, and the distance between the virtual object and the vibration device 32. The vibration sound determination unit 52 determines the vibration start time according to the playback start time of the sound associated with the virtual object. The vibration sound determination unit 52 determines the vibration pattern for each of the multiple vibration devices 32 based on the type of virtual object.

[0030] The vibration sound control unit 53 independently vibrates the multiple vibration devices 32 according to the vibration intensity determined by the vibration sound determination unit 52. The vibration sound control unit 53 may cause each of the multiple vibration devices 32 to reproduce the vibration pattern determined by the vibration sound determination unit 52. The vibration sound control unit 53 may output sound associated with a virtual object to headphones in synchronization with the reproduction of the vibration pattern.

[0031] The voice database 61 stores at least one voice data associated with a virtual object. The voice database 61 may also store voice data associated with a combination of a virtual object and a type of action. In the following, where it is written that a virtual object is "associated" or "linked" with a virtual object, a combination of a virtual object and a type of action may be used instead of a virtual object.

[0032] The vibration database 62 stores at least one vibration data associated with a virtual object. The vibration database 62 may also store vibration data associated with a combination of a virtual object and a type of motion.

[0033] The vibration conversion unit 55 generates vibration data based on audio data associated with a virtual object. The vibration data indicates a vibration pattern that causes each of the multiple vibration devices 32 to vibrate. The vibration conversion unit 55 converts the audio data stored in the audio database 61 into vibration data, and stores the converted vibration data in the vibration database 62, associating it with the information (e.g., virtual object) associated with the original audio data. The vibration conversion unit 55 may convert the audio data into vibration data using, for example, a low-pass filter.

[0034] Figure 4 is a flowchart showing an overview of the processing of the control device 10. First, the vibration conversion unit 55 included in the control device 10 converts audio data into vibration data using a low-pass filter (S101). More specifically, the vibration conversion unit 55 acquires audio data stored in association with each virtual object from the audio database 61, converts that audio data into vibration data using a low-pass filter, and stores the vibration data in association with the corresponding virtual object. This conversion makes it easier to intuitively grasp the correspondence between audio and vibration. The process in S101 may be performed only once at the beginning, or it may be performed beforehand.

[0035] The object control unit 51 then moves the virtual object in the virtual space (S102). In this process, the object control unit 51 determines the new position and orientation of the virtual object.

[0036] The vibration sound determination unit 52 determines the sound and vibration to be output based on the position of the virtual object (S103). Details of this process will be described later.

[0037] Once the sound and vibration to be output are determined, the vibration and sound control unit 53 controls the vibration device 32 based on that determination (S104), and controls the sound output device 40 based on that determination (S105). The processes in S104 and S105 are described in general terms, and the order of processing does not have to follow the description in Figure 4.

[0038] The control device 10 then determines whether the termination condition for the process is met (S106). If the termination condition is not met (N in S106), the time is advanced and the process from S102 onwards is repeated. If the termination condition is met (Y in S106), the process shown in Figure 4 is terminated. The termination condition may include receiving an input from the user indicating termination, or the elapsed of a predetermined time.

[0039] The processes from S103 to S105 will be explained in more detail. Figure 5 is a flowchart illustrating the processes related to the output of sound and vibration. The processes shown in Figure 5 are performed periodically for the virtual object that is the target of the processing.

[0040] First, the vibration sound determination unit 52 determines whether the current state satisfies the start conditions for sound and vibration (S201). These start conditions may be, for example, the virtual object starting a specific action, or the virtual object performing a specific action and the playback of sound data having finished.

[0041] If the starting conditions are met (Y in S201), the vibration sound determination unit 52 determines the magnitude of the vibration and the volume of the sound based on the position of the virtual object (S202).

[0042] Let's further explain how the magnitude of vibration is determined. Figure 6 is a diagram showing an example of the process for determining the magnitude of vibration, and it explains the process in S202 in more detail.

[0043] The vibration sound determination unit 52 acquires direction information and distance information for each vibration device 32 in the virtual space (S301). Direction information indicates the direction of the virtual object relative to the vibration device 32, and distance information indicates the distance between each vibration device 32 and the virtual object.

[0044] FIG. 7 is a diagram for explaining an example of a method for calculating the magnitude of vibration. FIG. 7 explains a method for acquiring the angle θ and the distance D for one of the plurality of vibration devices 32. In the virtual space, the plurality of vibration devices 32 are arranged along a circle (having a central axis C as the center) corresponding to the head circumference of the user's head 72 on a plane having the central axis C as a normal line. A representative position of the vibration device 32 may also lie on a circle centered on the central axis C, and the radius of the circle may be a predetermined value corresponding to the size of the head. This predetermined value may correspond to the head 72 defined by a sphere or a spheroid. Although FIG. 7 describes that the representative position of the vibration device 32 is the center of the vibration device 32, the representative position may be the innermost point of the vibration device 32 (the point where the vibration device 32 and the head 72 are in contact with each other). The position of each vibration device 32 in the virtual space is determined by the position and posture of the user's head.

[0045] A vector M extending from the central axis C to the center of a certain vibration device 32 corresponds to the main responsible direction of the vibration device 32. The target object 71 is a target for which the magnitude of vibration is determined among virtual objects. A line connecting the target object 71 and the vibration device 32 is provided, the length of the line is the distance D, and the direction of the line corresponds to direction information. The angle θ is calculated from the main responsible direction and the angle indicated by the direction information. The position of the virtual object when calculating the angle θ may be a position projected onto a plane that has the central axis C as a normal line and on which a circle corresponding to the head circumference of the user's head 72 exists.

[0046] When direction information and distance information are acquired for each of the plurality of vibration devices 32, the vibration sound determination unit 52 determines the intensity of vibration for each of the plurality of vibration devices 32 based on the main responsible direction, the direction information, and the information indicating the distance (S302). First, the vibration sound determination unit 52 obtains each θ from the main responsible direction and the direction information by the method described in the explanation of FIG. 7, and then determines the intensity of vibration using the angle θ and the distance D.

[0047] The magnitude of vibration (vibration intensity Ai) of the i-th vibration device 32 is calculated by the following formula. a and b may be arbitrary constants, and Dmax is the distance at which vibration becomes zero. This formula is an example for a case where the number of vibration devices 32 is eight.

[0048] According to this formula, the vibration intensity Ai changes smoothly according to a change in θ, reaching a maximum when the target object 71 is ahead in the primary responsible direction (θ=0), and becoming zero when the target object 71 is ahead in the primary responsible direction of the adjacent vibration device 32 (θ=45°). The closer the target object 71 is to the vibration device 32 or the head 72, the stronger the vibration becomes. When the target object 71 is located between the primary responsible directions of two vibration devices 32, the two vibration devices 32 vibrate, and the user can perceive a more precise direction of the target object 71 based on the difference in vibration intensity between the two vibration devices 32.

[0049] As described above, by using the direction and distance to the target object 71 and the primary responsible direction, the distance and direction of the target object 71 can be expressed by vibration intensity. The method for calculating the magnitude of vibration is not limited to the above formula; for example, the magnitude of vibration may be obtained by a formula using a continuous function different from a trigonometric function.

[0050] Note that the angle θ indicating direction and the distance D may be acquired by a simpler method. FIG. 8 is a diagram explaining another example of a method for calculating the magnitude of vibration. In the example of this diagram, instead of the line connecting the target object 71 and this vibration device 32, a line connecting the target object 71 and the center of the head 72 is used. The length of this line is the distance D, and the orientation of this line corresponds to direction information.

[0051] For example, in the case of headphones, the vibration and sound determining unit 52 determines the sound volume based on the positions of both ears and the position of the target object 71 in a virtual space. The sound volume may be determined by a known method.

[0052] Once the magnitude of each vibration of the vibration device 32 is determined, the vibration sound control unit 53 outputs the vibration data associated with the virtual object and the vibration magnitude to the vibration control circuit 31, and starts the playback of the vibration (S203). It is not necessary to output the vibration magnitude directly to the vibration control circuit 31. The vibration waveform to be output may be generated by the vibration control circuit 31 based on the vibration data and vibration magnitude, or the control device 10 may generate the vibration waveform from the vibration data and vibration magnitude using software and send a signal indicating that waveform to the vibration control circuit 31.

[0053] Furthermore, the vibration sound control unit 53 outputs the sound data associated with the virtual object and the sound volume to the sound control circuit 41, and starts sound playback (S204). It is not necessary to output the sound volume directly to the sound control circuit 41.

[0054] If the starting condition is not met in S201 (N), the processes from S202 to S204 are skipped.

[0055] The processing from S205 onwards is for changing the volume of vibrations and sounds in accordance with the movement of the virtual object while sound and vibration are being output.

[0056] The vibration-sound determination unit 52 determines whether it is currently outputting sound and vibration (S205). If it is outputting (Y in S205), the vibration-sound determination unit 52 re-determines the magnitude of the vibration and the volume of the sound based on the position of the virtual object (S206). The method for determining the magnitude of the vibration and the volume of the sound is the same as in S202, so the explanation is omitted.

[0057] Once the vibration magnitude of each of the multiple vibration devices 32 is determined, the vibration sound control unit 53 outputs the vibration magnitude to the vibration control circuit 31 to control the vibration of the vibration devices 32 (S207). The vibration sound control unit 53 may not directly output the vibration magnitude to the vibration control circuit 31, but instead control the vibration output by, for example, changing the vibration magnitude parameter in the process of generating the vibration waveform in software.

[0058] Furthermore, the vibration sound control unit 53 outputs the sound volume to the sound control circuit 41 and controls the sound output (S208). It is not necessary to output the sound volume directly to the sound control circuit 41.

[0059] If the starting condition is not met in S205 (N), the processes from S206 to S208 are skipped.

[0060] The processing from S205 to S208 ensures that vibrations and sounds are provided according to the latest position even when the virtual object is moving. For example, when a bee is moving, the location and magnitude of the vibration device 32 change in accordance with its movement, allowing the user to perceive the movement in ways other than sight.

[0061] In this embodiment, by changing the vibration intensity of each of the four or more vibration devices 32 according to the distance and direction of the virtual object, the user can intuitively perceive the position of the virtual object through touch. For example, it is possible to make objects around the user in the virtual space feel more realistic.

[0062] The methods for achieving this effect are not limited to those described above. For example, the tactile device 30 may be worn on a surface other than the head, and it does not have to be precisely ring-shaped. Also, the vibration data and audio data may be divided into multiple sub-data sets that are played back continuously. In this case, the processing from S205 to S208 is not performed, and the processing from S202 to S204 may be performed at the timing of the end of playback of one sub-data set and the start of playback of the next sub-data set.

Claims

1. A haptic presentation system comprising: a vibration determination means for determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device; and a control means for causing each of the at least four vibration devices to vibrate according to the determined intensity of vibration.

2. A tactile presentation system according to claim 1, wherein the vibration determination means determines the start time of the vibration according to the start time of playback of the sound associated with the virtual object.

3. A tactile presentation system according to claim 2, wherein the vibration determination means determines the vibration pattern of each of the at least four vibration devices based on the virtual object, and the control means causes each of the at least four vibration devices to reproduce the determined vibration pattern.

4. A tactile presentation system according to claim 3, further comprising a generation means for generating vibration patterns for each of the at least four vibration devices based on sounds associated with the virtual object.

5. A haptic presentation system according to claim 3, wherein the control means outputs sound associated with the virtual object to headphones in synchronization with the playback of the vibration pattern.

6. A tactile presentation method comprising: determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device; and vibrating each of the at least four vibration devices according to the determined intensity of vibration.

7. A program for causing a computer to function as a vibration determination means for determining the intensity of vibration for each of at least four vibration devices attached to a user at a distance from each other, based on information indicating the primary direction of the vibration device in a virtual space, the direction of a virtual object relative to the vibration device, and the distance between the virtual object and the vibration device; and a control means for causing each of the at least four vibration devices to vibrate according to the determined vibration intensity.

Images