Information processing device and information processing method

Abstract

This information processing device comprises a control unit. The control unit: acquires sensing data representing a nuance of a beat in a musical piece; generates, on the basis of the sensing data, a drive signal for a vibration device that causes a user to perceive the nuance of the beat as a sense of rhythm through vibration stimulation; mixes the drive signal and musical piece data of the musical piece so that the drive signal and the musical piece data can be synchronously output; and outputs the mixed musical piece data with vibration to the user.

Description

Information processing device and information processing method

[0001] This disclosure relates to an information processing device and an information processing method.

[0002] Conventionally, technologies for supporting the practice of playing musical instruments such as the piano have been known. For example, Patent Document 1 proposes a device that calculates the user's proficiency level based on at least one of the loudness and rhythm of the sound in musical instrument performance, and generates advice according to the calculated proficiency level, which is then presented to the user as visual or audible information.

[0003] Furthermore, in playing musical instruments, it is essential to intuitively and accurately grasp the "beat," which is the basic rhythmic guide and foundation of performance. When practicing musical pieces, developing a "sense of rhythm"—the ability to feel the beat of the music—is crucial for reproducing the characteristics of the piece as envisioned by composers, arrangers, and conductors.

[0004] Japanese Patent Publication No. 2020-013152

[0005] However, the conventional technologies described above still have room for improvement in effectively supporting the acquisition of a sense of rhythm in musical instrument playing.

[0006] For example, a metronome can represent the beat by ticking at regular intervals in time with the tempo of a song, but actual songs often have variations in the strength of the beats and fluctuations in the time intervals between beats. A metronome cannot represent these nuances of the beat.

[0007] Therefore, currently, instructors often use sounds, words, and actions to convey nuances of rhythm to their students. However, these current means of communication are insufficient to accurately convey the nuances of rhythm that instructors want to communicate to students in a way that students can perceive, i.e., as a sense of rhythm. Furthermore, the conventional techniques mentioned above merely present these means of communication by instructors as visual or auditory information.

[0008] Furthermore, these challenges are not limited to instrumental performances, but are common to all musical expressions, such as dancing or moving to the music.

[0009] Therefore, this disclosure proposes an information processing device and an information processing method that can effectively support the acquisition of a sense of rhythm for musical expression.

[0010] To solve the above problems, one form of information processing apparatus according to the present disclosure includes a control unit. The control unit acquires sensing data representing the nuances of beats in a musical piece, generates a drive signal for a vibration device that causes the user to perceive the nuances of beats as a sense of beat through vibration stimulation based on the sensing data, mixes the drive signal and the musical data of the musical piece so that they can be output synchronously, and outputs the mixed vibration-equipped musical data to the user.

[0011] This is a schematic diagram (1) of the beat sense acquisition support method according to the present disclosure. This is a schematic diagram (2) of the beat sense acquisition support method according to the present disclosure. This is a block diagram showing an example configuration of the beat sense acquisition support device according to the first embodiment. This is a flowchart (1) showing the processing procedure of the vibration data generation process according to the first embodiment. This is a flowchart (2) showing the processing procedure of the vibration data generation process according to the first embodiment. This is an explanatory diagram of the vibration data generation process according to the first embodiment. This is a diagram showing an example UI when playing music data with beat vibration. This is an explanatory diagram of adjusting the timing of vibration in relation to sound. This is a block diagram showing an example configuration of the beat sense acquisition support device according to the second embodiment. This is a diagram showing an example UI when sensing beat data by touching the screen. This is a flowchart (1) showing the processing procedure of the vibration data generation process according to the second embodiment. This is a flowchart (2) showing the processing procedure of the vibration data generation process according to the second embodiment. This is a diagram showing an example of beat vibration data generated by the vibration data generation process according to the second embodiment. This is a diagram (1) showing an example of visual representation feedback. This is a diagram (2) showing an example of visual representation feedback. This is a diagram (3) showing an example of visual representation feedback. This is Figure (4) showing an example of visual feedback. This is Figure (5) showing an example of visual feedback. This is a figure showing an example of a vibration device. This is a figure showing an example of selling music data and beat vibration data as a single music content. This is a figure showing a modified version that can be applied to live venues, etc. This is a figure showing a modified version that can be applied to sharing a sense of beat and tempo in ensemble playing. This is an explanatory diagram for checking the level of beat sense acquisition. This is a hardware configuration diagram showing an example of a computer that realizes the functions of a beat sense acquisition support device.

[0012] Embodiments of this disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0013] Furthermore, the following explanation will primarily use the case where the information processing device according to the embodiment of this disclosure (hereinafter referred to as "this embodiment") is a beat sensation acquisition support device 100 (see Figure 3) as the main example.

[0014] Furthermore, this disclosure will be explained in the following order of items: 1. Overview 2. First Embodiment 2-1. Configuration Example of a Beat Sense Acquisition Support Device According to the First Embodiment 2-2. Specific Example of Vibration Data Generation Processing According to the First Embodiment 2-3. Specific Example of UI During Playback of Music Data with Beat Vibration 3. Second Embodiment 3-1. Configuration Example of a Beat Sense Acquisition Support Device According to the Second Embodiment 3-2. Specific Example of UI During Beat Data Sensing by Screen Touch 3-3. Specific Example of Vibration Data Generation Processing According to the Second Embodiment 4. Modifications 4-1. First Modification 4-2. Second Modification 4-3. Third Modification 4-4. Fourth Modification 4-5. Fifth Modification 4-6. Sixth Modification 4-7. Other Modifications 5. Hardware Configuration 6. Conclusion

[0015] <<1. Overview>> Figure 1 is an overview diagram (part 1) of the beat sense acquisition support method according to the present disclosure. Figure 2 is an overview diagram (part 2) of the beat sense acquisition support method according to the present disclosure. The beat sense acquisition support method according to this embodiment is executed by the control unit 102 (see Figure 3) of the beat sense acquisition support device 100.

[0016] In general terms, the beat sense acquisition support method according to this embodiment is a technology that allows users to perceive the correct nuances of the beat of a musical piece as a sense of beat through vibrational stimulation. Hereinafter, "user" refers to a person who receives such vibrational stimulation, such as a trainee in musical instrument performance or dance acting, or a musician or dancer in an actual musical performance or dance acting setting. Furthermore, hereafter, composers, arrangers, conductors, instructors, choreographers, dance instructors, etc., in musical expression will be considered examples of "understanders" who correctly understand the characteristics of the nuances of the beat in a musical piece.

[0017] First, let's explain beat and sense of beat. In Figure 1, each note, labeled 1, 2, 3, ..., represents a beat. As shown in Figure 1 as "Beats that can be represented by a metronome," a metronome, for example, strikes a beat at regular intervals in time with the tempo of a piece of music. A sequence of beats forms a time signature. A metronome can also sound a small bell or similar at predetermined intervals, such as every two, three, four, or six beats. By matching this to the time signature, the user can perceive, for example, the beginning of a measure. In the following, "beat" may be referred to as "beat," and "sense of beat" as "sense of beat."

[0018] However, in actual musical expression such as playing an instrument, beats often fluctuate in temporal intervals, as shown in Figure 1 as "An example of actual musical beats." Also, as shown in the same figure by the difference in note size, beats often have dynamics. These nuances of beats, such as fluctuations and dynamics, manifest in actual musical expression as characteristics of the piece, such as intonation, tempo changes, momentum, dynamism, groove, and feel (hereinafter referred to as "feel" as appropriate).

[0019] Therefore, currently, instructors often use sounds, words, and actions to convey nuances of rhythm to users. However, these current means of communication are insufficient to accurately convey the nuances of rhythm that instructors want to communicate so that users can grasp them as an intuitive sense of rhythm.

[0020] Therefore, in the beat sense acquisition support method according to this embodiment, sensing data representing the nuances of the beat in a musical piece is acquired, a drive signal for a vibration device is generated based on the sensing data to make the user perceive the nuances of the beat as a sense of beat through vibration stimulation, the drive signal and the musical data of the musical piece are mixed so that they can be output synchronously, and the mixed music data with vibration is output to the user.

[0021] Let's take piano playing as an example. Specifically, as shown in Figure 2, in the rhythm sense acquisition support method according to this embodiment, the rhythm sense acquisition support device 100 first acquires "musical data". Musical data is data that records the sound source data of a performance by an instructor or other person or a musical piece.

[0022] Next, the rhythm sense acquisition support device 100 acquires "rhythm sensing data." The rhythm sensing data is, for example, data sensed as rhythm data when a teacher or the like plays musical data and strikes at least one key on the piano keyboard to express nuances of the beat, such as the strength and temporal fluctuations of the beat. While a teacher or the like can input nuances of the beat by striking at least one key, the device may be configured to allow input of nuances of the beat using multiple keys on the piano keyboard as needed.

[0023] The beat sense acquisition support device 100 then performs "vibration data generation" based on the beat sensing data and generates "beat vibration data". The beat vibration data is the drive signal for the vibration device 17 (see Figure 3), which allows the user to perceive the nuances of the beat of a musical piece as a sense of beat through vibration stimulation. Details of the "vibration data generation," that is, the vibration data generation process that generates beat vibration data, will be described later using Figures 4A and 4B, etc.

[0024] The beat sense acquisition support device 100 then "mixes" the music data and the beat vibration data to generate "music data with beat vibration." The music data with beat vibration is, for example, multi-channel data in which the sound of the first channel and the vibration of the second channel are synchronized. The term "mixing" here may be read as "mixing." By playing the vibration simultaneously with the sound using this music data with beat vibration, the user can perceive the correct nuances of the beat, which are difficult to accurately convey through sounds, words, actions, etc., as vibration stimuli. In other words, the beat sense acquisition support method according to this embodiment can effectively support the acquisition of a sense of beat in piano playing.

[0025] The following describes in detail an example of the configuration of a beat sense acquisition support device 100 to which the beat sense acquisition support method according to this embodiment is applied. In the following, the first embodiment will describe an example in which the above-mentioned piano keystrokes are sensed as beat data representing the beat. The second embodiment will describe an example in which screen touches on a smartphone 5 are sensed as beat data representing the beat.

[0026] <<2. First Embodiment>> <2-1. Example of Configuration of the Beat Sense Acquisition Support Device According to the First Embodiment> Figure 3 is a block diagram showing an example of the configuration of the beat sense acquisition support device 100 according to the first embodiment. Note that Figure 3 and Figure 8, shown later, show only the components necessary to explain the features of the embodiments of this disclosure, and descriptions of general components are omitted.

[0027] In other words, the components illustrated in Figures 3 and 8 are functional concepts and do not necessarily need to be physically configured as shown. For example, the specific forms of distribution and integration of each block are not limited to those shown, and it is possible to configure all or part of them by functionally or physically distributing and integrating them in any unit according to various loads and usage conditions.

[0028] Furthermore, in explanations using Figure 3 or Figure 8, explanations of components that have already been described may be simplified or omitted.

[0029] As shown in Figure 3, the beat sense acquisition support device 100 has a storage unit 101 and a control unit 102. The beat sense acquisition support device 100 is also connected to a sound source input unit 11, a beat sensor 13, a speaker 15, a vibration device 17, and a UI (User Interface) unit 19.

[0030] The sound source input unit 11 is a device that inputs sound sources for musical data. When recording a performance by an instructor or other person to create musical data, the sound source input unit 11 is, for example, a microphone. When using pre-recorded sound source data to create musical data, the sound source input unit 11 is, for example, an input interface for a recording medium on which the sound source data is stored.

[0031] The beat sensor 13 is a sensor that senses beat data. In the first embodiment, the beat sensor 13 is a key strike sensor. As described above, the key strike sensor is provided so as to be able to sense the key strike data of at least one key on the piano keyboard, which is struck by an instructor or the like, as beat data representing the beat, when acquiring beat sensing data.

[0032] The speaker 15 outputs sound during playback of music data or music data with beat vibration. The vibration device 17 outputs vibration during playback of music data with beat vibration. The vibration device 17 is composed of a vibrator (not shown) and is provided so as to be wearable by the user. The UI unit 19 provides a UI mainly related to the display operation of application software (hereinafter referred to as "app" as appropriate) provided by the beat sense acquisition support device 100. The UI unit 19 is implemented by, for example, a touch panel display. The speaker 15, vibration device 17, and UI unit 19 may be integrated into a smartphone 5 wirelessly connected to the beat sense acquisition support device 100.

[0033] The storage unit 101 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory, or by disk devices such as hard disk drives and optical disc drives. In the example shown in Figure 3, the storage unit 101 stores music data DB (Database) 101a, beat sensing data DB 101b, beat vibration data DB 101c, and music data DB 101d with beat vibration.

[0034] The music data DB101a is a database that stores the music data shown in Figure 2. The beat sensing data DB101b is a database that stores the beat sensing data shown in Figure 2.

[0035] The beat vibration data DB101c is a database that stores the beat vibration data shown in Figure 2. The beat vibration-equipped music data DB101d is a database that stores the beat vibration-equipped music data shown in Figure 2.

[0036] The control unit 102 corresponds to a so-called processor or controller. The control unit 102 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 102 executes the program according to the illustrated exemplary embodiment stored in the storage unit 101, using the RAM as a work area. Also, the control unit 102 can be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0037] The control unit 102 includes a music data recording unit 102a, a beat data sensing unit 102b, a vibration data generation unit 102c, a mixing unit 102d, and an output control unit 102e, and realizes or executes the information processing functions and operations described below. Also, although not shown in the figure, the control unit 102 reads and executes the program of the app dedicated to the beat acquisition support device 100 stored in the storage unit 101, and provides the functions of such an app via the UI unit 19.

[0038] The music data recording unit 102a acquires and records the music data shown in FIG. 2 via the sound source input unit 11. Also, the music data recording unit 102a stores the recorded music data in the music data DB 101a.

[0039] The beat data sensing unit 102b senses the beat sensing data shown in FIG. 2 via the beat sensor 13. In the first embodiment, the beat data sensing unit 102b acquires, as beat data representing a beat, the key strike data sensed for at least one key of the piano keyboard described above. Also, the beat data sensing unit 102b stores the acquired beat data in the beat sensing data DB 101b as beat sensing data.

[0040] When the beat data sensing unit 102b senses beat data, the output control unit 102e extracts the target music data from the music data DB 101a and outputs it from the speaker 15. The instructor or the like presses the above-mentioned keys while listening to the music data output from the speaker 15.

[0041] The vibration data generation unit 102c extracts the beat sensing data to be processed from the beat sensing data DB 101b, executes vibration data generation processing based on this, and generates the beat vibration data shown in FIG. 2. The details of the vibration data generation processing will be described later. Also, the vibration data generation unit 102c stores the generated beat vibration data in the beat vibration data DB 101c.

[0042] The mixing unit 102d extracts the music data and the beat vibration data to be mixed from the music data DB 101a and the beat vibration data DB 101c, respectively, mixes them, and generates the music data with beat vibration shown in FIG. 2. Also, the mixing unit 102d stores the generated music data with beat vibration in the music data with beat vibration DB 101d.

[0043] The output control unit 102e causes the UI unit 19 to output the UI screen of the dedicated application provided by the control unit 102. Also, the output control unit 102e extracts the music data with beat vibration specified by the user via the UI unit 19 and targeted for playback from the music data with beat vibration DB 101d, outputs the sound from the speaker 15, and at the same time outputs the vibration from the vibration device 17.

[0044] <2-2. Specific example of vibration data generation processing according to the first embodiment> Next, a specific example of the vibration data generation processing according to the first embodiment will be described. FIG. 4A is a flowchart (part 1) showing the processing procedure of the vibration data generation processing according to the first embodiment. Also, FIG. 4B is a flowchart (part 2) showing the processing procedure of the vibration data generation processing according to the first embodiment. Also, FIG. 5 is an explanatory diagram of the vibration data generation processing according to the first embodiment.

[0045] Here, we will use the example of striking the keys for one measure of a 3 / 4 time signature. Furthermore, we will assume that the resonant frequency of the oscillator in the vibration device 17 is around 110 Hz, which is the frequency at which it vibrates most strongly.

[0046] The vibration data generation unit 102c first receives the beat sensing data to be processed, i.e., the keystroke waveform data, from the beat sensing data DB 101b (step S101). Then, the vibration data generation unit 102c extracts data with a keystroke depth of 4 mm or more from the input waveform data (step S102). Data with a depth of less than 4 mm is removed as a preliminary step. For an example of the data before extraction, please refer to the waveform for "keystroke depth" in Figure 5, and for examples of the data after extraction, please refer to the waveforms for "keystroke speed" and "key release speed" in Figure 5, respectively.

[0047] The vibration data generation unit 102c then calculates, for each beat of the three beats, the keystroke time, the keystroke depth, the time when the keystroke depth is maximum, the keystroke speed, and the key release speed in the extracted data (step S103). For examples of keystroke speed and key release speed, please refer to the bar graphs for "Keystroke Speed" and "Key Release Speed" in Figure 5, respectively.

[0048] Then, the vibration data generation unit 102c reflects the keystroke time and keystroke depth in the time and amplitude of a sine wave (see the lower diagram in Figure 5) for each beat (step S104).

[0049] Next, as shown in Figure 4B, the vibration data generation unit 102c determines whether the key release speed is 4 mm or less for each beat (step S105). If the key release speed is 4 mm or less (step S105, Yes), the vibration data generation unit 102c sets the vibration frequency corresponding to the beat to 80 Hz (step S106).

[0050] If the key release speed exceeds 4 mm (Step S105, No), the vibration data generation unit 102c determines whether the key release speed is 6 mm or less (Step S107). If the key release speed is 6 mm or less (Step S107, Yes), the vibration data generation unit 102c sets the frequency corresponding to the beat to 120 Hz (Step S108). Since 120 Hz is close to the resonance frequency of around 110 Hz, the vibration data generation unit 102c reduces the amplitude to 0.5 times to avoid strong vibration stimulation (Step S109).

[0051] If the key release speed exceeds 6 mm (step S107, No), the vibration data generation unit 102c determines whether the key release speed is 8 mm or less for each beat (step S110). If the key release speed is 8 mm or less (step S110, Yes), the vibration data generation unit 102c sets the frequency corresponding to the corresponding beat to 160 Hz (step S111). If the key release speed exceeds 8 mm (step S110, No), the vibration data generation unit 102c sets the frequency corresponding to the corresponding beat to 180 Hz (step S112).

[0052] Next, the vibration data generation unit 102c determines whether the keystroke speed for each beat is 11 mm or less, which is a predetermined threshold (step S113). If the keystroke speed is 11 mm or less (step S113, Yes), the vibration data generation unit 102c sets the amplitude of the sine wave to be smaller in proportion to the keystroke speed (step S114).

[0053] If the keystroke speed exceeds 11 mm (step S113, No), the vibration data generation unit 102c determines whether the keystroke speed is 18 mm or less for each beat (step S115). If the keystroke speed is 18 mm or less (step S115, Yes), the vibration data generation unit 102c replaces 110 Hz by 0.01 s at the time when the keystroke depth is maximum for the corresponding beat (step S116).

[0054] If the keystroke speed exceeds 18 mm (step S115, No), the vibration data generation unit 102c replaces the keystroke depth with 110 Hz for 0.03 s at the time when the keystroke depth is maximum in the corresponding beat (step S117). The reason for replacing it with 110 Hz is to make it perceived by the user as the strongest vibration stimulus. For an example of replacing with 110 Hz, please refer to the sine wave corresponding to the first beat in the lower diagram of Figure 5.

[0055] The vibration data generation unit 102c then outputs sine wave vibration data, i.e., pulse vibration data, and stores it in the pulse vibration data DB 101c, thereby ending the vibration data generation process according to the first embodiment.

[0056] As a result, as shown in Figure 5, for the first beat, which has a deeper keystroke depth, longer keystroke duration, faster keystroke speed, and slower release speed than the second and third beats, the following beat vibration data is generated. Specifically, for the first beat, beat vibration data is generated that has strength and weight, starting with a strong vibration stimulus of 110 Hz, which is a strong beat, followed by vibrations with an amplitude of 80 Hz. This beat vibration data for the first beat provides the user with a vibration stimulus that makes them feel the driving force of the music.

[0057] On the other hand, for the second and third beats, where the keystroke depth is shallower, the keystroke time is shorter, the keystroke speed is slower, and the key release speed is faster than for the first beat, the generated beat vibration data has a higher frequency, shorter vibration time, and narrower amplitude than for the first beat. This beat vibration data for the second and third beats provides a vibration stimulus that the user perceives as sharp.

[0058] Note that the values ​​shown in Figures 4A, 4B, and 5 are merely examples and may be changed as appropriate depending on the performance of the vibration device 17, for example.

[0059] <2-3. Specific Examples of UI When Playing Music Data with Beat Vibration> Next, specific examples of the UI when playing music data with beat vibration in the application provided by the beat sense acquisition support device 100 according to this embodiment will be explained using Figures 6 and 7. Figure 6 is a diagram showing an example of the UI when playing music data with beat vibration. Figure 7 is an explanatory diagram of the timing adjustment of vibration in relation to sound.

[0060] When playing music data with beat vibration, the control unit 102 provides a UI screen via the UI unit 19, for example, as shown in Figure 6. On this UI screen, the Play buttons corresponding to "Music," "Haptic," and "Music + Haptic" allow for standalone playback of ch1 (sound), standalone playback of ch2 (vibration), and simultaneous playback of ch1 and ch2, respectively.

[0061] Furthermore, the "Vol." button shown in area R1 allows for individual volume adjustment of both sound and vibration. Additionally, the slider bar B1 and button B2 in "Haptic delay Tuning" shown in area R2 allow for adjustment of the timing of vibration relative to sound, making it "earlier" or "slower."

[0062] Slide bar B1 allows the user to adjust the timing of vibrations in response to sound by sliding it to the "faster" or "slower" side, depending on the amount of slide. Button B2 allows the user to adjust the timing of vibrations in response to sound by adjusting it in the smallest increments, either "faster" or "slower," each time the button B2 is touched.

[0063] As shown in Figure 7, adjusting the timing of vibrations in relation to sound to be "earlier" means that the timing of the vibrations in channel 2 is earlier than the timing of the sound in channel 1. Conversely, adjusting the timing of vibrations in relation to sound to be "slower" means that the timing of the vibrations in channel 2 is later than the timing of the sound in channel 1. These adjustments can be made by storing the data of the channel to be delayed in a memory area such as the memory unit 101, for example, designated as a "Delay Unit," thereby adjusting the amount of delay.

[0064] Furthermore, in the musical expression of a piece of music, it is expected that the appropriate timing for both the music data and the beat vibration data may differ depending on the purpose and characteristics of the expression, the style of the music, the tempo, etc. The UI examples explained using Figures 6 and 7 are useful for adjusting the timing in such cases.

[0065] <<3. Second Embodiment>> Next, we will describe a second embodiment that differs from the first embodiment described above, in which screen touches on the smartphone 5 are sensed as beat data representing a beat.

[0066] <3-1. Example of Configuration of the Beat Sense Acquisition Support Device According to the Second Embodiment> Figure 8 is a block diagram showing an example of the configuration of the beat sense acquisition support device 100 according to the second embodiment. Note that Figure 8 corresponds to Figure 3 already shown. Therefore, the explanation using Figure 8 will mainly focus on the differences from Figure 3.

[0067] As shown in Figure 8, the beat sense acquisition support device 100 according to the second embodiment can be integrated as a smartphone 5 together with the sound source input unit 11, beat sensor 13, speaker 15, vibration device 17, and UI unit 19.

[0068] In the example shown in Figure 8, the sound source input unit 11 can be implemented using the microphone and recording medium input interface installed in the smartphone 5. Also, in the example shown in Figure 8, the beat sensor 13 can be implemented using the three-axis accelerometer installed in the smartphone 5.

[0069] Furthermore, in the example shown in Figure 8, the speaker 15 can be realized by a speaker mounted on the smartphone 5. Also, in the example shown in Figure 8, the vibration device 17 can be realized by a vibrator mounted on the smartphone 5. Also, in the example shown in Figure 8, the UI section 19 can be realized by a touch panel display mounted on the smartphone 5.

[0070] Furthermore, in the example shown in Figure 8, the memory unit 101 can be implemented by a memory device mounted on the smartphone 5. Also, in the example shown in Figure 8, the control unit 102 can be implemented by a CPU, MPU, GPU, etc., mounted on the smartphone 5.

[0071] <3-2. Specific Examples of UI for Pulse Data Sensing via Screen Touch> Next, we will explain specific examples of UI for pulse data sensing via screen touch. Figure 9 shows an example of UI for pulse data sensing via screen touch.

[0072] During beat data sensing via screen touch, the control unit 102 provides a UI screen via the UI unit 19, for example, as shown in Figure 9. On this UI screen, when the user touches the beat data sensing selection button B3, which is labeled "Beat Recording," a UI screen including a beat data sensing start button B4, labeled "Start Beat Recording," is displayed.

[0073] Then, when the user touches the start button B4, a heart rate data sensing screen including a touch area TP1 and a stop button B5 is displayed. When the user touches the touch area TP1 on this heart rate data sensing screen, the heart rate data sensing unit 102b senses the screen touch data as heart rate data.

[0074] Then, when the user touches the beat data sensing stop button B5, which is labeled "Stop Beat Recording," the beat data sensing unit 102b stops sensing beat data.

[0075] <3-3. Specific Examples of Vibration Data Generation Processing According to the Second Embodiment> Next, specific examples of vibration data generation processing according to the second embodiment will be described. Figure 10A is a flowchart (part 1) showing the processing procedure of vibration data generation processing according to the second embodiment. Figure 10B is a flowchart (part 2) showing the processing procedure of vibration data generation processing according to the second embodiment. Figure 11 is a diagram showing an example of beat vibration data generated by vibration data generation processing according to the second embodiment. Here, as with the first embodiment, a screen touch for one measure of 3 / 4 time will be used as an example.

[0076] The vibration data generation process according to the second embodiment generally involves sensing screen touches using an acceleration sensor and recording the sum of the touch time and the absolute value of the maximum three-axis acceleration within the touch time for each beat. Then, beat vibration data is generated by setting the amplitude of a predetermined vibration pattern to increase as the sum increases. The amplitude of the vibration pattern is set in six steps, for example, taking into consideration the so-called Weber-Fechner law.

[0077] As shown in Figure 10A, when an instructor or the like presses the aforementioned start button B4 (step S201), the vibration data generation unit 102c starts measuring the acceleration in the x, y, and z axes using the beat sensor 13 (step S202).

[0078] The vibration data generation unit 102c then determines whether or not the instructor or other person has tapped the screen (step S203). If the screen has been tapped (step S203, Yes), the vibration data generation unit 102c records the screen touch time and the sum of the absolute values ​​of the maximum three-axis accelerations within the touch time (step S204), and repeats the process from step S203.

[0079] On the other hand, if the screen is not tapped (step S203, No), the vibration data generation unit 102c determines whether or not the instructor or other person pressed the stop button B5 (step S205). If the stop button B5 was not pressed (step S205, No), the vibration data generation unit 102c repeats the process from step S203.

[0080] If the stop button B5 is pressed (step S205, Yes), the vibration data generation unit 102c takes into account a touch delay of 80 msec and creates a vibration time array for specifying the vibration pattern (step S206).

[0081] Next, as shown in Figure 10B, the vibration data generation unit 102c generates data when the sum of the three-axis accelerations is 5 m / s². ２Determine whether it is greater (step S207). If it is greater (step S207, Yes), the vibration data generation unit 102c sets, for example, the amplitude of the vibration pattern to 255 (step S208) and transitions to step S218.

[0082] Determine whether the sum of the three-axis accelerations is greater than 5 m / s ２ In the following case (step S207, No), the vibration data generation unit 102c determines whether the sum of the three-axis accelerations is greater than 4 m / s ２ (step S209). If it is greater (step S209, Yes), the vibration data generation unit 102c sets, for example, the amplitude of the vibration pattern to 171 (step S210) and transitions to step S218.

[0083] Determine whether the sum of the three-axis accelerations is greater than 4 m / s ２ In the following case (step S209, No), the vibration data generation unit 102c determines whether the sum of the three-axis accelerations is greater than 3 m / s ２ (step S211). If it is greater (step S211, Yes), the vibration data generation unit 102c sets, for example, the amplitude of the vibration pattern to 114 (step S212) and transitions to step S218.

[0084] Determine whether the sum of the three-axis accelerations is greater than 3 m / s ２ In the following case (step S211, No), the vibration data generation unit 102c determines whether the sum of the three-axis accelerations is greater than 2 m / s ２ (step S213). If it is greater (step S213, Yes), the vibration data generation unit 102c sets, for example, the amplitude of the vibration pattern to 77 (step S214) and transitions to step S218.

[0085] Determine whether the sum of the three-axis accelerations is greater than 2 m / s ２ In the following case (step S213, No), the vibration data generation unit 102c determines whether the sum of the three-axis accelerations is greater than 1 m / s ２ (step S215). If it is greater (step S215, Yes), the vibration data generation unit 102c sets, for example, the amplitude of the vibration pattern to 51 (step S216) and transitions to step S218.

[0086] The sum of the three-axis accelerations is 1 m / s². ２ In the following case (step S215, No.), the vibration data generation unit 102c sets the amplitude of the vibration pattern to 34, for example (step S217). Then, the vibration data generation unit 102c creates a vibration amplitude array for specifying the vibration pattern (step S218), and terminates the vibration data generation process according to the second embodiment.

[0087] The vibration data generation unit 102c executes the processing procedures shown in Figures 10A and 10B to generate, for example, beat vibration data as shown in Figure 11. In the example in Figure 11, sensing starts at time T1, a screen tap corresponding to the first beat occurs at time T2, and the finger is lifted at time T3. Based on the tap time and 3-axis acceleration of this screen tap, the first beat vibration data is generated with the vibration time (Time) set to "257", which is the time from time T2 to time T3, and the amplitude (Amp) set to "255".

[0088] In the example shown in Figure 11, a screen tap corresponding to the second beat occurs at time T4, and the finger is released at time T5. Based on the tap time and 3-axis acceleration of this screen tap, second beat vibration data is generated with a vibration time of "66" (the time from time T4 to time T5) and an amplitude of "77".

[0089] In the example shown in Figure 11, a screen tap corresponding to the third beat occurs at time T6, and the finger is lifted at time T7. Based on the tap time and 3-axis acceleration of this screen tap, the vibration data for the third beat is generated with a vibration time of "65" (the time from time T6 to time T7) and an amplitude of "51". Note that during the periods when no screen taps occur from time T1 to time T2, from time T3 to time T4, and from time T5 to time T6, a vibration-free period with an amplitude of "0" is set for each.

[0090] In this way, it is possible to generate beat vibration data that expresses nuances of the beat, etc., by tapping the screen.

[0091] Note that the values ​​shown in Figures 10A, 10B, and 11 are merely examples and may be changed as appropriate depending on the performance of the smartphone 5, for example.

[0092] <<4. Modifications>> <4-1. First Modification> Now, several modifications can be given to each of the embodiments described above. For example, in the embodiments described above, examples were given in which vibrations that convey a sense of beat are fed back to the user at the same time as sound, but it is also possible to provide visual representation as feedback.

[0093] Figures 12 to 16 are diagrams (1) to (5) showing examples of visual feedback. As an example of presenting visual representations to the user along with sound and vibration, for example, as shown in Figure 12, one beat may be represented by a ball on the UI unit 19, and the size of the ball may represent the strength of the beat. Furthermore, the sense of beat may be conveyed to the user by an animation display in which this ball bounces, changes size, and moves position in accordance with the temporal changes of one measure.

[0094] The animation shown in Figure 12 will be drawn based, for example, on the correspondence between the animation elements shown in Figure 13 and the beat sensing data, as well as the equations of motion.

[0095] Furthermore, as an alternative representation of the example in Figure 12, for example, as shown in Figure 14, an indicator ID 1 that moves in accordance with the temporal change of one measure may be displayed, and an animation display in which a ball representing the beat bounces in accordance with the movement of this indicator ID 1 may be performed. In this case, the amount the ball bounces may depend on the amplitude time of each beat in the beat vibration data. In other words, to summarize Figures 12 to 14, any visual representation that expresses the fluctuations in the strength and temporal interval of the beat in accordance with the temporal change of one measure is acceptable.

[0096] Furthermore, as an example of other visual representations, as shown in Figure 15, the UI unit 19 may be used to display the trajectory of the conductor's baton in accordance with the temporal changes of one measure, and an animation may be performed to highlight the trajectory corresponding to the strong beats, such as the second and fourth beats, by using a more prominent color than the other beats.

[0097] Furthermore, as shown in Figure 16, for example, a rectangle consisting of a series of band-shaped regions corresponding to each of the 1st to 4th beats in 4 / 4 time can be displayed on the UI unit 19, and an animation display can be performed in which the band-shaped regions corresponding to the 2nd and 4th beats, which are to be designated as strong beats, are highlighted by changing color over time. Note that the examples shown in Figures 15 and 16 are both examples in which the so-called off-beats are emphasized in 4 / 4 time.

[0098] Furthermore, when performing musical expressions such as playing an instrument or dancing, visual representations alone, as shown in Figures 12 to 16, have difficulty conveying the correct nuances of the beat, a problem inherent in existing technologies. However, by using cross-modal effects in conjunction with vibration, it becomes possible to convey the nuances of the beat more accurately.

[0099] <4-2. Second Modification> Next, Figure 17 shows an example of the vibration device 17. Incidentally, in order to provide vibration feedback to the user, it is desirable that the vibration device 17 can reliably deliver vibration stimulation to the user even while the user is playing a musical instrument or dancing.

[0100] In that sense, the vibration device 17 may be, for example, a band-type vibration device 17a that can be worn by the user on their wrist or ankle, as shown in Figure 17. Alternatively, the vibration device 17 may be a neck speaker-type vibration device 17b (see Figure 20) that can be worn by the user around their neck. Furthermore, the vibration device 17 may be a vibration device 17c (see Figure 20) mounted on a smartwatch.

[0101] In addition, the vibration device 17 may be a ring type that can deliver vibrations to the fingertips, for example, for a pianist. Alternatively, the vibration device 17 may be a cushion embedded in the seat on which a pianist sits while playing. Furthermore, when used in conjunction with visual representations, the vibration device 17 may vibrate the entire screen.

[0102] <4-3. Third Modification> Next, Figure 18 shows an example of selling song data and beat vibration data as a single music content. As shown in Figure 18, the beat vibration data generated using the beat sense acquisition support method according to this embodiment may be sold to the user as a music content set with song data. In this case, the user can learn the rhythm of the song data before the live performance, for example, via their own smartphone 5. In addition, artists who provide music content can communicate the intended rhythm of the song to the user in advance of the live performance. This is expected to further increase the sense of unity at the actual live venue.

[0103] <4-4. Fourth Modification> Next, Figure 19 shows a modification applicable to live venues, etc. For live venues, for example, the aforementioned band-type vibration device 17a, etc., may be distributed to the audience, either for a fee or free of charge, and vibration instructions may be given via wireless control, etc., using beat vibration data that matches the clapping or rhythm desired by the artist.

[0104] At this time, the sounds of the performance emitted from the stage will reach the audience seats in order of proximity to the stage. Accordingly, the aforementioned vibration instructions may be transmitted with a delay according to the distance from the stage.

[0105] For example, as shown in Figure 19, suppose that the first seating area R11 is closest to the stage, the third seating area R13 is furthest from the stage, and the second seating area R12 is located between the first and third seating areas R11 and R13. In this case, the performance control device that transmits vibration instructions to the vibration device 17 delays the vibration instructions for the second seating area R12 from those for the first seating area R11, according to the speed of sound. Similarly, the performance control device delays the vibration instructions for the third seating area R13 from those for the second seating area R12.

[0106] This allows audience members far from the stage to receive rhythmic vibrations corresponding to the sound arriving from the stage, even at large-scale festivals held in large venues. This can reduce, for example, the time lag in groove and is expected to increase the sense of unity.

[0107] Furthermore, regarding live venues, the visual representations shown in Figures 12 to 16 may be displayed on a large screen SC1 or similar installed at the venue, as shown in Figure 19. In such cases, even users who are rhythmically challenged or those new to live concerts can easily understand the clapping and groove that the artist wants, not only through vibration but also through visual representations, which is expected to further enhance the sense of unity at the live venue.

[0108] <4-5. Fifth Modification> Next, Figure 20 shows a modification applied to sharing a sense of beat and tempo in an ensemble performance. In this modification, broadcast audio, one of the functions of "LE Audio," a Bluetooth® voice communication standard, can be used.

[0109] Broadcast audio is a feature that allows audio to be broadcast from one audio source to multiple audio syncs within a certain range, similar to radio broadcasting.

[0110] For example, in ensemble performances such as those in orchestras, discrepancies in the timing of each instrument's performance can be fatal. Therefore, broadcast audio may be used to transmit vibrations that convey a reference beat and tempo to vibration devices 17 worn by multiple musicians.

[0111] Specifically, as shown in Figure 20, the control unit 102 first senses the conductor's movements and generates a reference beat and tempo (step S11). Then, the control unit 102 broadcasts the reference beat and tempo as vibration signals (timing, intensity) to each device (vibration devices 17a, 17b, 17c, smartphone 5) (step S12).

[0112] Then, each device vibrates, reproducing the beat and tempo (step S13). This allows the conductor and musicians to share a reference beat and tempo in an ensemble setting.

[0113] Furthermore, since each instrument has its own characteristics, such as wind instruments having a time lag before sound is produced compared to other instruments, it is desirable that the timing of vibration signal distribution be controlled according to the characteristics of each instrument. For example, the timing of vibrations in wind instruments may be adjusted to be earlier to compensate for the time lag compared to other instruments.

[0114] Furthermore, the sharing method shown in Figure 20 can also be applied to sharing groove in group dance performances, for example. In this case, each dancer receives vibration signals generated by sensing the movements of, for example, a choreographer or dance instructor, through their respective vibrating devices. This allows everyone to synchronize their rhythm and sense of speed, thereby sharing the groove. Moreover, by aligning everyone's groove, a sense of unity is created, and the dance performance intended by the choreographer or dance instructor can be reproduced, thereby increasing the overall quality of the work.

[0115] Furthermore, while not limited to group dance performances, it is also possible to learn the groove of professional dancers by receiving vibration signals generated from sensing their movements with various vibrating devices. In this case, the vibration signals would transmit the tempo, intensity, pauses, and speed of the professional dancers' movements as vibrational stimuli. As a result, dancers can expect effects such as developing a deeper connection to the music, creating sharper movements, and training a sense of rhythm that can adapt to any song.

[0116] For applications in dance performance, it is desirable to use, for example, a band-type vibration device 17a that can be worn on the ankle so that feedback can be provided to each dancer's feet, or to configure the floor on which each dancer performs their dance as a vibration device 17, and provide feedback by transmitting vibrations from the floor to each dancer.

[0117] <4-6. Sixth Modification> Next, Figure 21 is an explanatory diagram for checking the level of beat sense acquisition. In each of the embodiments described above, beat vibration data was generated based on input from an instructor, etc., but the level of beat sense acquisition may also be checked using beat vibration data generated based on the user's own input.

[0118] Specifically, as shown in Figure 21, for example, before training, the user creates musical data based on their own performance and beat vibration data based on their own input. Then, during training, the user trains using training data consisting of musical data performed by an instructor or other person and beat vibration data input by the instructor or other person.

[0119] After training, the user creates new music data based on their own performance and beat vibration data based on their input. The similarity of the beat vibration data before and during training, and during and after training, is then calculated using, for example, an AI (Artificial Intelligence) model.

[0120] This allows users to check their own level of mastery of rhythm by comparing the calculated similarity scores. The rhythm-sense acquisition support device 100 may also provide the user with a function to check this level of mastery of rhythm, for example, as part of the aforementioned application.

[0121] <4-7. Other Modifications> Furthermore, among the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0122] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, the beat sense acquisition support device 100 may be distributed and configured in multiple devices or systems according to the functions performed by each processing unit 102a to 102e of the control unit 102.

[0123] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate.

[0124] <<5. Hardware Configuration>> Furthermore, the beat sense acquisition support device 100 according to the embodiment of the present disclosure described above is realized by a computer 1000 having a configuration such as that shown in Figure 22. Figure 22 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the beat sense acquisition support device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, and input / output interface 1600. The parts of the computer 1000 are connected by a bus 1050.

[0125] The CPU 1100 operates based on programs stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0126] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0127] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that stores a program according to the embodiment of this disclosure, which is an example of program data 1450.

[0128] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0129] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as microphones and touch panels via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as displays and speakers via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0130] For example, when the computer 1000 functions as a beat sense acquisition support device 100 according to the embodiment of this disclosure, the CPU 1100 of the computer 1000 realizes the functions of the control unit 102 by executing a program loaded on the RAM 1200. The secondary storage device 1400 stores the program according to this disclosure and data in the storage unit 101. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as an alternative example, these programs may be obtained from other devices via an external network 1550.

[0131] Furthermore, not limited to the hardware configuration described with reference to Figure 22, the functions realized by the components described herein may be implemented in a circuit configuration or processing circuit configuration, including a general-purpose processor, application-specific processor, integrated circuit, ASICs (Application Specific Integrated Circuits), CPU, conventional circuit, and / or a combination thereof, programmed to realize the described functions. The processor includes transistors and other circuits and is considered a circuit configuration or processing circuit configuration. The processor may be a programmed processor that executes a program stored in memory. For example, the processor implements the functions of the control unit 102 as a programmed processor.

[0132] In this specification, circuit configurations, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0133] If the hardware is a processor that is considered to be a type of circuit configuration, then the circuit configuration, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0134] <<6. Conclusion>> As described above, according to one embodiment of the present disclosure, the beat sense acquisition support device 100 (corresponding to an example of an "information processing device") includes a control unit 102. The control unit 102 acquires sensing data representing the nuances of beat in a musical piece, generates a drive signal for a vibration device that causes the user to perceive the nuances of beat as a sense of beat through vibration stimulation based on the sensing data, mixes the drive signal and the musical data of the piece so that they can be output synchronously, and outputs the mixed vibration-equipped musical data to the user. This makes it possible to effectively support the acquisition of a sense of beat when making musical expressions.

[0135] Although the embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0136] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0137] Furthermore, this technology can also be configured as follows: (1) An information processing device comprising a control unit that acquires sensing data representing the nuances of beat in a musical piece, generates a drive signal for a vibration device that causes the user to perceive the nuances of beat as a sense of beat through vibration stimulation based on the sensing data, mixes the drive signal and the musical data of the musical piece so that they can be output synchronously, and outputs the mixed vibration-equipped musical data to the user. (2) The information processing device according to (1), wherein the nuances of beat include at least the strength of the beat and fluctuations in the temporal interval of the beat. (3) The information processing device according to (1) or (2), wherein the control unit acquires the sensing data by sensing an action performed by a person who understands the characteristics of the nuances of beat on an instrument to express the nuances of beat. (4) The information processing device according to (3), wherein the instrument is a keyboard instrument, and the control unit acquires the sensing data by sensing a keystroke performed by the person who understands at least one key on the keyboard of the keyboard instrument. (5) The information processing apparatus according to (4), wherein the control unit generates the drive signal based on the keystroke time, keystroke depth, time when the keystroke depth is maximum, keystroke speed, and key release speed calculated from the sensing data. (6) The information processing apparatus according to (5), wherein the control unit reflects the keystroke time and keystroke depth in the time and amplitude of the sine wave that becomes the drive signal. (7) The information processing apparatus according to (6), wherein the control unit generates the drive signal such that the frequency increases as the key release speed increases. (8) The information processing apparatus according to (7), wherein the control unit replaces the frequency at least at the time when the keystroke depth is maximum with the frequency at which the vibration is maximum when the keystroke speed exceeds a predetermined threshold. (9) The information processing apparatus according to (8), wherein the control unit reduces the amplitude of the sine wave according to the keystroke speed when the keystroke speed is below the threshold.(10) The information processing apparatus according to (1) or (2), wherein the control unit acquires the sensing data by sensing a screen touch on a touch panel display that expresses the nuance of the beat, performed by a person who understands the nuance of the beat. (11) The information processing apparatus according to (10), wherein the control unit senses the screen touch using an acceleration sensor, records the sum of the touch time at the screen touch corresponding to each beat and the absolute value of the maximum three-axis acceleration within the touch time, and generates the drive signal by setting the amplitude of a predetermined vibration pattern to increase as the sum increases. (12) The information processing apparatus according to any one of (1) to (11), wherein the control unit presents a visual representation to the user in accordance with the output of the drive signal. (13) The information processing apparatus according to (12), wherein the visual representation is an animation display that expresses the strength of the beat over time changes over at least one measure. (14) The information processing apparatus according to (13), wherein the animation display is displayed on a large screen installed at a live venue, and the animation operates in synchronization with the output of the drive signal to the vibration device held by the audience at the live venue. (15) The information processing apparatus according to (14), wherein the drive signal to the vibration device is output with a delay according to the distance from the stage at the live venue. (16) An information processing method performed by the information processing apparatus, comprising: acquiring sensing data representing the nuance of beats in a musical piece; generating a drive signal for a vibration device that causes the user to perceive the nuance of beats as a sense of beats through vibration stimulation, based on the sensing data; mixing the drive signal and the musical data of the musical piece so that they can be output in synchronization; and outputting the mixed vibration-equipped musical data toward the user.

[0138] 5 Smartphone 11 Sound source input unit 13 Beat sensor 15 Speaker 17 Vibration device 100 Beat sense acquisition support device 101 Storage unit 101a Music data DB 101b Beat sensing data DB 101c Beat vibration data DB 101d Music data DB with beat vibration 102 Control unit 102a Music data recording unit 102b Beat data sensing unit 102c Vibration data generation unit 102d Mixing unit 102e Output control unit SC1 Large screen

Claims

1. An information processing device comprising: a control unit that acquires sensing data representing the nuances of beat in a musical piece; generates a drive signal for a vibration device that causes the user to perceive the nuances of beat as a sense of beat through vibration stimulation, based on the sensing data; mixes the drive signal and the musical data of the musical piece so that they can be output synchronously; and outputs the mixed vibration-equipped musical data to the user.

2. The information processing apparatus according to claim 1, wherein the nuance of the beat includes at least the intensity of the beat and fluctuations in the temporal interval of the beat.

3. The information processing apparatus according to claim 1, wherein the control unit acquires the sensing data by sensing an action performed by a person who understands the nuances of the beat on an instrument that expresses the nuances of the beat.

4. The information processing apparatus according to claim 3, wherein the instrument is a keyboard instrument, and the control unit acquires the sensing data by sensing a keystroke made by the user on at least one key of the keyboard of the keyboard instrument.

5. The information processing apparatus according to claim 4, wherein the control unit generates the drive signal based on the keystroke time, keystroke depth, the time when the keystroke depth is maximized, the keystroke speed, and the key release speed calculated from the sensing data.

6. The information processing apparatus according to claim 5, wherein the control unit reflects the keystroke time and the keystroke depth in the time and amplitude of the sine wave that becomes the drive signal.

7. The information processing apparatus according to claim 6, wherein the control unit generates the drive signal such that the frequency increases as the speed of key release increases.

8. The information processing apparatus according to claim 7, wherein the control unit replaces the frequency at least at the time when the keystroke depth is maximum with the frequency at which the vibration is maximum when the keystroke speed exceeds a predetermined threshold.

9. The information processing apparatus according to claim 8, wherein the control unit reduces the amplitude of the sine wave in accordance with the speed of the keystroke when the speed of the keystroke is less than or equal to the threshold.

10. The information processing apparatus according to claim 1, wherein the control unit acquires the sensing data by sensing a screen touch on a touch panel display that expresses the nuance of the beat, performed by a person who understands the nuance of the beat.

11. The information processing apparatus according to claim 10, wherein the control unit senses the screen touch using an acceleration sensor, records the sum of the touch time and the absolute value of the maximum three-axis acceleration within the touch time corresponding to the beat, and generates the drive signal by setting the amplitude of a predetermined vibration pattern to increase as the sum increases.

12. The information processing apparatus according to claim 1, wherein the control unit presents a visual representation corresponding to the drive signal to the user in accordance with the output of the drive signal.

13. The information processing apparatus according to claim 12, wherein the visual representation is an animation display that expresses the strength of beats in accordance with temporal changes over at least one measure.

14. The information processing apparatus according to claim 13, wherein the animation display is shown on a large screen installed at a live venue, and the animation operates in synchronization with the output of the drive signal to the vibration device held by the audience at the live venue.

15. The information processing apparatus according to claim 14, wherein the drive signal to the vibration device is output with a delay according to the distance from the stage of the live venue.

16. An information processing method performed by an information processing device, comprising: acquiring sensing data representing the nuances of beats in a musical piece; generating a drive signal for a vibration device that causes a user to perceive the nuances of beats as a sense of beats through vibrational stimulation, based on the sensing data; mixing the drive signal and musical data of the musical piece so that they can be output synchronously; and outputting the mixed vibration-equipped musical data to the user.

Images