Sound-effect-synchronized vibration control method, system, electronic device, and storage medium
By extracting psychological loudness features from audio data and generating relative frequency and intensity curves, the problem of heavy workload and poor user experience for sound effect vibration designers is solved, achieving efficient sound effect vibration feedback control and improving the user's sensory experience.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- AAC ACOUSTIC TECH (SHANGHAI) CO LTD
- Filing Date
- 2024-12-13
- Publication Date
- 2026-06-18
Smart Images
Figure CN2024139163_18062026_PF_FP_ABST
Abstract
Description
A method, system, electronic device, and storage medium for controlling vibration in response to sound effects. Technical Field
[0001] This application relates to the field of haptic feedback technology, and in particular to a method, system, electronic device, and storage medium for controlling vibration in response to sound effects. Background Technology
[0002] With the development of technology, the application of vibration feedback is no longer limited to simple vibration prompts. Currently, commonly used electronic devices such as mobile phones, tablets, and even game controllers, which are the most common carriers for video and game content providers, are gradually incorporating vibration feedback into their audio and video content, thereby bringing users a more diverse sensory experience. However, high-quality vibration feedback now requires vibration designers to manually add vibration effects that match the scene to the content. As the demand for vibration feedback in audio and video increases, the workload of vibration designers also increases.
[0003] Furthermore, according to relevant psychological research, human subjective auditory perception is not linear. The loudness value of general audio data cannot directly map people's subjective auditory perception. The vibration feedback mechanism currently provided by related technologies cannot guarantee the consistency between human hearing and touch, resulting in a poor vibration feedback experience and making it difficult to bring users an immersive sensory experience.
[0004] Therefore, it is necessary to provide an automatically generated method for controlling vibrations in response to sound effects, in order to balance the workload of relevant designers and effectively improve the human body's experience of sound vibration feedback mechanisms.
[0005] [Summary of the Invention] Technical issues
[0006] The purpose of this application is to provide a method, device, electronic device and storage medium for controlling vibrations based on sound effects, which can at least solve the problems of heavy workload for sound effect vibration designers and poor human experience of sound effect vibration feedback mechanisms in related technologies. Technical solutions
[0007] To address the aforementioned technical problems, the first aspect of this application provides a method for controlling vibration in response to sound effects, comprising:
[0008] Extract the psychological loudness features of the target audio data; wherein, the psychological loudness features include: the loudness value, instantaneous power peak, center frequency, and time of a single frame of audio data;
[0009] Intensity information is calculated using the loudness value and the instantaneous power peak, and the center frequency is mapped to the center Mel frequency;
[0010] A relative frequency curve is obtained based on the center Mel frequency of all the single-frame audio data, and a relative intensity curve is obtained based on the intensity information of all the single-frame audio data;
[0011] A vibration signal that follows the sound effect is generated based on the relative frequency curve and the relative intensity curve.
[0012] The vibration signal that follows the sound effect is sent to the actuator to drive the actuator to output corresponding vibration feedback.
[0013] A second aspect of this application provides a sound-effect vibration control device, comprising:
[0014] The feature extraction module is used to extract the psychological loudness features of the target audio data; wherein, the psychological loudness features include: the loudness value, instantaneous power peak, center frequency, and frame time of a single frame of audio data;
[0015] The feature mapping module is used to calculate intensity information using the loudness value and the instantaneous power peak, and to map the center frequency to the center Mel frequency;
[0016] The parameter determination module is used to obtain a relative frequency curve based on the center Mel frequency of all the single-frame audio data, and to obtain a relative intensity curve based on the intensity information of all the single-frame audio data;
[0017] The signal generation module is used to generate a vibration signal that follows the sound effect based on the relative frequency curve and the relative intensity curve;
[0018] The driving module is used to send the vibration signal that follows the sound effect to the actuator to drive the actuator to output corresponding vibration feedback.
[0019] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the sound-effect vibration control method described in the first aspect of the embodiments of this application.
[0020] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the sound-effect vibration control method described in the first aspect of the embodiments of this application. Beneficial effects
[0021] As can be seen from the above description, compared with related technologies, the beneficial effects of this application are as follows:
[0022] First, psychological loudness features are extracted from the target audio data, and intensity information is calculated using the loudness value and instantaneous power peak value from the psychological loudness features, and the center frequency is mapped to the center Mel frequency. Second, relative frequency curves are obtained based on the center Mel frequencies of all the single-frame audio data, and relative intensity curves are obtained based on the intensity information of all the single-frame audio data. Finally, a sound-effect vibration signal is generated based on the relative frequency curves and the relative intensity curves, and the sound-effect vibration signal is sent to the actuator to drive the actuator to output corresponding vibration feedback. As can be seen from the above, the embodiments of this application extract psychological loudness features from audio data, use psychological evaluation indicators combined with Mel weighting to convert loudness values, center frequencies, and power peak values into relative intensity curves and relative frequency curves, and generate sound-effect vibration signals based on the relative intensity curves and relative frequency curves, so as to retain effective information to the maximum extent and map it to the actuator vibration parameters, thereby completing sound-effect vibration feedback control. This can effectively improve the richness of human body's perception of vibration feedback and bring users an immersive sensory experience. In addition, this method automatically generates vibration feedback, which is simple and efficient. On the one hand, it can meet the needs of content manufacturers for independent batch conversion, and on the other hand, only a small amount of optimization by designers is required to complete high-quality vibration feedback, which can further balance the workload of designers.
[0023] It should be understood that the description in this section is not intended to identify key or important features of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0024] To more clearly illustrate the related technologies or the technical solutions in the embodiments of this application, the drawings used in the description of the related technologies or the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application, and not all embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 is a schematic flowchart of the vibration control method based on sound effects provided in an embodiment of this application;
[0026] Figure 2 is a detailed flowchart of the vibration control method based on sound effects provided in the embodiments of this application;
[0027] Figure 3 is a schematic diagram of the vibration signal generated based on the original sound effect in an embodiment of this application;
[0028] Figure 4 is a schematic diagram of the program modules of the sound effect vibration control device provided in the embodiment of this application;
[0029] Figure 5 is a block diagram of the electronic device provided in an embodiment of this application;
[0030] Figure 6 is a block diagram of a computer-readable storage medium provided in an embodiment of this application. Embodiments of the present invention
[0031] To make the objectives, technical solutions, and advantages of this application more apparent and understandable, this application will be clearly and completely described below in conjunction with its embodiments and accompanying drawings. Throughout, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. It should be understood that the various embodiments of this application described below are merely illustrative of this application and are not intended to limit this application. That is, all other embodiments obtained by those skilled in the art based on the various embodiments of this application without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.
[0032] Please refer to Figure 1, which is a schematic flowchart of the vibration control method following the sound effect provided in the embodiment of this application. The vibration control method following the sound effect includes the following steps 101 to 105.
[0033] Step 101: Extract the psychological loudness features of the target audio data.
[0034] In this embodiment of the application, if you want to generate the final vibration signal that follows the sound effect, you need to first obtain the target audio data of the audio and extract the psychological loudness features from the obtained target audio data. The psychological loudness features include the loudness value, instantaneous power peak, center frequency and time of a single frame of audio data.
[0035] In this application embodiment, the psychological evaluation index is mainly reflected in the loudness value of the single-frame audio data. Generally, audio data has two main characteristics: sound pressure level (SPL) and frequency. However, according to relevant research in psychoacoustics, these two physical characteristics do not directly map to people's subjective auditory perception; loudness is the subjectively perceived intensity of sound. Loudness is usually measured in phons or sones. Loudness typically increases with increasing SPL, but the relationship is not linear, and the human ear's perception of loudness gradually weakens as SPL increases. Different frequencies of sound, at the same SPL, are perceived differently by the human ear. Generally, the human ear is more sensitive to mid-to-high frequency sounds and less sensitive to low-frequency sounds.
[0036] Therefore, in this embodiment, before extracting the psychological loudness features of the target audio data, it is necessary to first read in the original audio data with sound effects and preprocess the audio, including using appropriate filters for mid-frequency processing, so that the vibrations generated by the sound effects are more in line with the human tactile perception threshold, ultimately achieving a better vibration feedback effect. Prior to this, the preprocessing also includes downsampling the audio. First, a low-pass filter is used to filter the signal, removing high-frequency components to prevent aliasing during downsampling. Then, the filtered signal is downsampled to reduce the amount of audio data.
[0037] Step 102: Calculate intensity information using loudness value and instantaneous power peak, and map the center frequency to the center Mel frequency.
[0038] In this embodiment, after obtaining the psychological loudness characteristics, the characteristic information needs to be converted into parameters to be considered for specific vibrations, namely vibration intensity and vibration frequency. Among the aforementioned characteristic information, the loudness value and the instantaneous power peak value can characterize the intensity of the audio. Therefore, in this embodiment, corresponding weights are set for the loudness value and the instantaneous power peak value, and then the weighted value is obtained, which is the intensity information that can characterize the vibration intensity.
[0039] Similarly, the center frequency in the above-mentioned feature information can represent the frequency information of the vibration. However, based on the above psychological research, audio signals are composed of a series of periodic vibrations with different frequencies. The human ear has different perceptions of sounds with different frequencies and can distinguish different pitches by frequency. However, the human ear's perception of frequency is not linear, but exhibits nonlinear characteristics.
[0040] Therefore, in this embodiment, to ensure consistency between auditory and tactile perception, the center frequency is mapped to a center Mel frequency. In one embodiment, a Mel scale is introduced to convert the center frequency into a center Mel frequency. The Mel scale is a non-linear frequency scale used to convert frequency into Mel values. The conversion between the Mel scale and frequency can be obtained through a formula, and the Mel frequency refers to the frequency value obtained after conversion by the Mel scale. Compared with linear frequencies, Mel frequencies are more consistent with human hearing. There are various methods to convert general linear frequencies into Mel frequencies that are consistent with human hearing, including using Mel scale filter banks, Mel frequency cepstral coefficients (MFCC), etc. This application does not limit these methods to a single method; those skilled in the art can flexibly set the parameters based on the embodiments of this application and the actual application scenario.
[0041] Step 103: Obtain the relative frequency curve based on the center Mel frequency of all single-frame audio data, and obtain the relative intensity curve based on the intensity information of all single-frame audio data.
[0042] In this embodiment, after extracting the psychological loudness features of the audio data, it is necessary to map the features to the vibration parameters of the actuator. Therefore, after obtaining the center Mel frequency of the single-frame audio data, it is also necessary to obtain a relative frequency curve based on the center Mel frequency. The horizontal axis represents the frame number, and the vertical axis represents the relative frequency, establishing the correspondence between the relative frequency and the frame number, that is, characterizing the change of audio frequency with the frame number.
[0043] Specifically, firstly, the relative frequency range of the actuator used to generate vibration feedback is obtained. Then, based on a preset first nonlinear function, the center Mel frequency of all the single-frame audio data is mapped to the relative frequency range to obtain a relative frequency curve. The first nonlinear function is a custom-constructed nonlinear function used to convert the audio frequency into the relative frequency of the actuator's vibration.
[0044] In this embodiment of the application, it is also necessary to obtain a relative intensity curve based on the intensity information, with the horizontal axis representing the number of frames and the vertical axis representing the relative intensity, which establishes the correspondence between the relative intensity and the number of frames.
[0045] Specifically, firstly, the relative intensity range of the actuator used to generate vibration feedback is obtained. Then, based on a preset second nonlinear function, the intensity information of all the single-frame data is mapped to the relative intensity range to obtain an initial relative intensity curve. Further, the initial relative intensity curve is optimized based on a human-perceptible vibration intensity threshold to obtain the final relative intensity curve. The second nonlinear function is a custom-constructed nonlinear function used to convert audio intensity information into the relative intensity of vibration generated by the actuator.
[0046] Step 104: Generate a vibration signal that follows the sound effect based on the relative frequency curve and the relative intensity curve.
[0047] Specifically, after obtaining the relative frequency curve and the relative intensity curve, the relative frequency corresponding to the frame time can be found on the relative frequency curve, and the relative intensity corresponding to the frame time can be found on the relative intensity curve. Then, the relative intensity, relative frequency and the corresponding frame time are integrated to generate a vibration signal that follows the sound effect.
[0048] In one optional implementation of this embodiment, in order to expand the application range of the sound-effect vibration signal, the final parameters can be written into a standard vibration file, and then the standard vibration file can be parsed using a parsing tool to generate the corresponding sound-effect vibration signal. By adjusting the parameters of the parsing tool, the generated vibration feedback can be applied to different vibration devices and different scenarios.
[0049] Step 105: Send the vibration signal along with the sound effect to the actuator to drive the actuator to output the corresponding vibration feedback.
[0050] In one optional embodiment of this example, the step of obtaining a relative frequency curve based on the center Mel frequency of all the single-frame audio data includes: obtaining a relative frequency range for the actuator used to generate vibration feedback; mapping the center Mel frequency of all the single-frame audio data to the relative frequency range based on a preset first nonlinear function to obtain a relative frequency curve; the step of obtaining a relative intensity curve based on the intensity information of all the single-frame audio data includes: obtaining a relative intensity range for the actuator used to generate vibration feedback; mapping the intensity information of all the single-frame audio data to the relative intensity range based on a preset second nonlinear function to obtain a relative intensity curve.
[0051] Specifically, in practical applications, actuators used to generate vibration feedback have different operating frequencies and intensities. Therefore, it is necessary to first obtain the relative frequency range and relative intensity range of the actuator used to generate vibration feedback. The center Mel frequency of a single frame of audio data is then mapped to the relative frequency range using a preset first nonlinear function to obtain a relative frequency curve. Similarly, the intensity information of a single frame of audio data is mapped to the relative intensity range using a preset second nonlinear function to obtain a relative intensity curve. Furthermore, the relative frequency curve and relative intensity curve are used to characterize the audio data, and in subsequent steps, these curves are used to generate a vibration signal that follows the sound effect, thus achieving vibration that follows the sound effect.
[0052] Furthermore, in an optional embodiment of this example, the method further includes: mapping the relative intensity curve and the relative frequency curve to a vibration sensing curve based on a preset vibration sensing function; recording points in the vibration sensing curve that are below the lower limit value perceptible to the human body as first deletion points based on the lower limit value of the human body vibration sensing function; obtaining the first corresponding time of all the first deletion points, and deleting the points in the relative intensity curve and the relative frequency curve that are at the first corresponding time to obtain the optimized relative intensity curve and relative frequency curve.
[0053] Specifically, in practical applications, the vibration feedback generated by some actuators is actually difficult for the human body to perceive. Therefore, the vibration feedback can be optimized based on the lower limit of the human body's perceptible vibration intensity. A pre-set fitted vibration sensing function is used to map the current relative intensity curve and relative frequency curve to a vibration sensing curve according to f(intensity, frequency). Points in the vibration sensing curve that are lower than the lower limit of the human body's perceptible vibration intensity are recorded as deletion points, and the corresponding points in the relative intensity curve and relative frequency curve at the corresponding frame time are deleted accordingly. This yields the optimized relative intensity curve and relative frequency curve, thereby optimizing the vibration feedback mechanism that follows the sound effect.
[0054] Furthermore, in an optional embodiment of this example, the step of mapping the relative intensity curve and the relative frequency curve to a vibration sensing curve based on a preset vibration sensing function includes: obtaining relative intensity information contained in the relative intensity curve; performing weighted processing on the relative intensity information to obtain a weighted relative intensity curve; and mapping the relative frequency curve and the weighted relative intensity curve to a vibration sensing curve based on the preset vibration sensing function.
[0055] Specifically, in practical applications, although the vibration feedback generated by some actuators can be perceived by the human body, it is not obvious enough. Some different vibration feedbacks cannot be significantly distinguished by the human body. Therefore, it is necessary to weight the vibration intensity to increase the intensity contrast, that is, to make the original strong vibration stronger and the original weak vibration weaker. In subsequent steps, vibrations below the lower limit of human-perceptible vibration intensity are deleted, ultimately improving the human body's experience of the sound effect vibration feedback mechanism.
[0056] Furthermore, in an optional embodiment of this example, the method further includes: performing adjacent value verification on the target points in the vibration curve; if the target points in the vibration curve meet the preset conditions, then recording the target points as second deletion points; obtaining the second corresponding time of all the second deletion points, and deleting the points in the optimized relative intensity curve and the optimized relative frequency curve that are at the second corresponding time to obtain the simplified relative intensity curve and the simplified relative frequency curve.
[0057] Specifically, in this embodiment, in addition to optimizing the vibration effect, long signals can also be simplified. It is understood that when performing adjacent value verification on target points in the vibration sensing curve, for any point in the vibration sensing curve, if that point, along with its predecessor and successor points, can jointly describe the local characteristics of the curve within a preset range, then that point can be recorded as a deletion point. The points corresponding to the deletion points in the optimized relative intensity curve and the optimized relative frequency curve are then omitted, ultimately completing the simplification of the long signal. This simplification not only reduces the workload of relevant designers but also ensures the continuity of the vibration experience.
[0058] In one optional embodiment of this example, the step of generating a sound-effect vibration signal based on the relative frequency curve and the relative intensity curve includes: generating an initial sound-effect vibration signal based on the relative frequency curve and the relative intensity curve; adding a short signal to the initial sound-effect vibration signal according to preset short signal parameters to obtain a final sound-effect vibration signal; wherein, the short signal parameters include the short signal addition time, the short signal frequency, and the short signal intensity.
[0059] Specifically, in practical applications, to further enrich the layering of vibration intensity and optimize the human experience of sound effect vibration feedback, short signals can be added to key locations of the sound effect to enhance vibration feedback. First, an initial vibration signal following the sound effect is generated based on the relative frequency curve and the relative intensity curve. Then, short signals are added to the initial vibration signal following the sound effect based on preset short signal parameters or short signal parameters generated based on real-time audio signals to obtain the final vibration signal following the sound effect.
[0060] Furthermore, in an optional embodiment of this example, the method further includes: filtering the peak points of the relative intensity curve to obtain a first peak point; fitting all the instantaneous power peaks into an instantaneous power peak curve, and obtaining a target point in the instantaneous power peak curve corresponding to the first peak point based on preset filtering conditions; and verifying the peak position of the first peak point and the target point to obtain the short signal parameters.
[0061] Furthermore, in an optional embodiment of this example, the step of filtering peak points of the relative intensity curve to obtain a first peak point includes: filtering peak points of the relative intensity curve to obtain initial peak points; connecting all the initial peak points and smoothing them to obtain a relative intensity envelope curve; and filtering peak points of the relative intensity envelope curve to obtain a first peak point.
[0062] Furthermore, in an optional embodiment of this example, the step of fitting all the instantaneous power peaks into an instantaneous power peak curve and obtaining the target point corresponding to the first peak point in the instantaneous power peak curve based on preset filtering conditions includes: obtaining the corresponding time of all the first peak points and combining it with a preset minimum time difference to obtain all corresponding time intervals; fitting all the instantaneous power peaks into an instantaneous power peak curve and obtaining all points in the instantaneous power peak curve that are within the corresponding time interval; and recording all the points within the corresponding time interval as target points corresponding to the first peak point.
[0063] Specifically, in this embodiment, the peak points of the relative intensity curve are first screened and the corresponding times of the peak points are obtained. Then, a minimum time difference is preset to obtain the corresponding time interval. Finally, the peak points of the relative intensity curve at the specified times and the points within the corresponding time intervals of the instantaneous power peak curve are checked for peak positions according to a preset method. If the point at the corresponding time is determined to be a peak point, a short signal is added at that point. Thus, by adding short signals at key positions of the sound effect, the sense of layering in the vibration feedback effect is enhanced.
[0064] Furthermore, in an optional embodiment of this example, the step of verifying the peak position of the first peak point and the target point to obtain the short signal parameters includes: filtering the peak points of the instantaneous power peak curve to obtain a second peak point; verifying whether the second peak point exists in the target point; if the second peak point exists in the target point, determining that the first peak point and the target point have passed the peak position verification, and recording the verified first peak point as the target first peak point; recording the corresponding time of the target first peak point as the short signal addition time; obtaining the relative frequency corresponding to the short signal addition time on the relative frequency curve to obtain the short signal frequency; obtaining the relative intensity corresponding to the short signal addition time on the relative intensity curve, and increasing the relative intensity according to a preset relative intensity increase value to obtain the short signal intensity; and obtaining the short signal parameters based on all the short signal addition times, the short signal frequency, and the short signal intensity.
[0065] Specifically, in this embodiment, if the peak point at time t also has a peak within the corresponding time interval of the instantaneous power peak curve, a short signal is added at this point, and the corresponding time is recorded as the short signal addition time. At the same time, the relative frequency corresponding to the short signal addition time on the relative frequency curve is obtained to obtain the short signal frequency. The relative intensity corresponding to the short signal addition time on the relative intensity curve is obtained, and the relative intensity is increased according to a preset relative intensity increase value to obtain the short signal intensity. Finally, the short signal parameters can be obtained by combining the short signal addition time, the short signal frequency, and the short signal intensity.
[0066] As can be seen from the above, the embodiments of this application extract psychological loudness features from audio data, and use psychological evaluation indicators combined with Mel weighting to convert loudness values, center frequencies, and peak power into relative intensity curves and relative frequency curves. Based on these curves, a vibration signal following the sound effect is generated to maximize the retention of effective information mapped to actuator vibration parameters, thus completing vibration feedback control following the sound effect. This effectively enhances the richness of human perception of vibration feedback, providing users with an immersive sensory experience. Furthermore, this method automatically generates vibration feedback, is simple and efficient, and can meet the needs of content providers for independent batch conversion. On the other hand, only minor optimization by designers is required to achieve high-quality vibration feedback, further balancing the workload of designers.
[0067] In summary, the detailed process of the vibration control method based on sound effects in this application can be referred to Figure 2, and is as follows:
[0068] Step 201: Read in the original audio data with sound effects, and preprocess the original audio data to obtain the target audio data;
[0069] Step 202: Extract the psychological loudness features of the target audio data;
[0070] Step 203: Calculate intensity information using the loudness value and instantaneous power peak value in the psychological loudness characteristics, and map the center frequency to the center Mel frequency;
[0071] Step 204: Obtain the relative frequency curve based on the center Mel frequency of all single-frame audio data, and obtain the relative intensity curve based on the intensity information of all single-frame audio data;
[0072] Step 205: Optimize the relative intensity curve and relative frequency curve based on the human-perceptible vibration intensity;
[0073] Step 206: Simplify the optimized relative intensity curve and relative frequency curve;
[0074] Step 207: Generate an initial vibration signal following the sound effect based on the simplified relative intensity curve and relative frequency curve;
[0075] Step 208: Add a short signal to the initial sound effect vibration signal to obtain the final sound effect vibration signal;
[0076] Step 209: Send the final vibration signal that accompanies the sound effect to the actuator to drive the actuator to output the corresponding vibration feedback.
[0077] For a more detailed process of each step in steps 201 to 209, please refer to the relevant sections shown above. The embodiments of this application will not be repeated here.
[0078] It should be understood that the sequence number of each step in this embodiment does not imply the order in which the steps are executed. The execution order of each step should be determined by its function and internal logic, and should not constitute a unique limitation on the implementation process of this application embodiment.
[0079] Please refer to Figure 3, which is a schematic diagram of the vibration signal generated based on the original sound effect in the embodiment of this application. It can be clearly seen that the vibration signal obtained by the vibration control method based on the sound effect in the embodiment of this application can retain the effective information to the motor vibration parameters to the maximum extent. Long signals are simplified, and short signals are added to the key positions of the sound effect to enrich the sense of vibration intensity. It is user-friendly for designers while ensuring the continuity of vibration, and finally completes the conversion from sound effect to vibration.
[0080] Please refer to Figure 4, which shows a sound-effect vibration control device provided in an embodiment of this application. This device can be used to implement the sound-effect vibration control method involved in this embodiment. The sound-effect vibration control device mainly includes:
[0081] The feature extraction module 401 is used to extract the psychological loudness features of the target audio data; wherein, the psychological loudness features include: the loudness value, instantaneous power peak, center frequency and frame time of a single frame audio data;
[0082] The feature mapping module 402 is used to calculate intensity information using the loudness value and the instantaneous power peak, and to map the center frequency to the center Mel frequency;
[0083] The parameter determination module 403 is used to obtain a relative frequency curve based on the center Mel frequency of all the single-frame audio data, and to obtain a relative intensity curve based on the intensity information of all the single-frame audio data;
[0084] Signal generation module 404 is used to generate a vibration signal that follows the sound effect based on the relative frequency curve and the relative intensity curve;
[0085] The drive module 405 is used to send the sound effect vibration signal to the actuator to drive the actuator to output corresponding vibration feedback.
[0086] In some embodiments of this example, the parameter determination module is specifically used to: obtain the relative frequency range of the actuator used to generate vibration feedback; map the center Mel frequency of all the single-frame audio data to the relative frequency range based on a preset first nonlinear function to obtain a relative frequency curve; obtain the relative intensity range of the actuator used to generate vibration feedback; and map the intensity information of all the single-frame audio data to the relative intensity range based on a preset second nonlinear function to obtain a relative intensity curve.
[0087] Furthermore, in some embodiments of this example, the parameter determination module is also used to: map the relative intensity curve and the relative frequency curve to a vibration sensing curve based on a preset vibration sensing function; record points in the vibration sensing curve that are lower than the lower limit value based on the human body's perceptible vibration intensity as first deletion points; obtain the first corresponding time of all the first deletion points, and delete the points in the relative intensity curve and the relative frequency curve that are at the first corresponding time to obtain the optimized relative intensity curve and relative frequency curve.
[0088] Furthermore, in some embodiments of this example, when the parameter determination module performs the function of mapping the relative intensity curve and the relative frequency curve to a vibration sensing curve based on a preset vibration sensing function, it is specifically used to: obtain the relative intensity information contained in the relative intensity curve; perform weighted processing on the relative intensity information to obtain a weighted relative intensity curve; and map the relative frequency curve and the weighted relative intensity curve to a vibration sensing curve based on the preset vibration sensing function.
[0089] Furthermore, in some embodiments of this example, the parameter determination module is also used to: perform adjacent value verification on the target point in the vibration curve; if the target point in the vibration curve can be characterized by the value of its adjacent point, then record the target point as the second deletion point; obtain the second corresponding time of all the second deletion points, and delete the points in the optimized relative intensity curve and the optimized relative frequency curve that are at the second corresponding time to obtain the simplified relative intensity curve and the simplified relative frequency curve.
[0090] In some embodiments of this example, the signal generation module is specifically used to: generate an initial sound-effect vibration signal based on the relative frequency curve and the relative intensity curve; add a short signal to the initial sound-effect vibration signal according to preset short signal parameters to obtain the final sound-effect vibration signal; wherein, the short signal parameters include the short signal addition time, the short signal frequency, and the short signal intensity.
[0091] Furthermore, in some embodiments of this example, the signal generation module is also used to: filter peak points on the relative intensity curve to obtain a first peak point; connect all the first peak points and smooth them to obtain a relative intensity envelope curve; filter peak points on the relative intensity envelope curve to obtain a second peak point; obtain the corresponding time of all the second peak points and combine them with a preset minimum time difference to obtain all corresponding time intervals; fit all the instantaneous power peaks into an instantaneous power peak curve and obtain all points in the instantaneous power peak curve that are within the corresponding time interval; and perform peak position verification between the second peak points and the points within the corresponding time interval to obtain the short signal parameters.
[0092] Furthermore, in some embodiments of this example, when the signal generation module performs the function of verifying the peak position of the second peak point with the points within the corresponding time interval to obtain the short signal parameters, it is specifically used to: filter the peak points of the instantaneous power peak curve to obtain a third peak point; if the third corresponding time of the third peak point is within the corresponding time interval, record the third corresponding time as the short signal addition time; obtain the relative frequency corresponding to the short signal addition time on the relative frequency curve to obtain the short signal frequency; obtain the relative intensity corresponding to the short signal addition time on the relative intensity curve, and increase the relative intensity according to a preset relative intensity increase value to obtain the short signal intensity; and obtain the short signal parameters based on all the short signal addition times, the short signal frequency, and the short signal intensity.
[0093] In one optional embodiment of this invention, the sound-effect vibration control device further includes a preprocessing module. The preprocessing module is used to: read in the original audio data with sound effects, and preprocess the original audio data to obtain target audio data.
[0094] According to the sound-effect vibration control device provided in this embodiment, psychological loudness features of target audio data are extracted; intensity information is calculated using the loudness value and the instantaneous power peak, and the center frequency is mapped to the center Mel frequency; relative frequency curves are obtained based on the center Mel frequencies of all single-frame audio data, and relative intensity curves are obtained based on the intensity information of all single-frame audio data; a sound-effect vibration signal is generated based on the relative frequency curves and the relative intensity curves; the sound-effect vibration signal is sent to the actuator to drive the actuator to output corresponding vibration feedback. Through the implementation of this application, psychological loudness features are extracted from audio data, and a psychological evaluation index combined with Mel weighting is used to convert loudness values, center frequencies, and power peaks into relative intensity curves and relative frequency curves. A sound-effect vibration signal is generated based on the relative intensity curves and relative frequency curves to maximize the retention of effective information mapped to the actuator vibration parameters, thus completing the sound-effect vibration feedback control. This effectively improves the richness of human perception of vibration feedback, bringing users an immersive sensory experience. In addition, this method automatically generates vibration feedback, which is simple and efficient. On the one hand, it can meet the needs of content manufacturers for independent batch conversion, and on the other hand, only a small amount of optimization by designers is required to complete high-quality vibration feedback, which can further balance the workload of designers.
[0095] Please refer to Figure 5, which is a block diagram of the electronic device provided in the embodiments of this application.
[0096] As shown in Figure 5, this application embodiment also provides an electronic device that can be used to implement the sound effect vibration control method in the foregoing embodiment. The electronic device includes a memory 501 and at least one processor 502. The memory 501 is used to store at least one program, and when the at least one program is executed by the at least one processor 502, the at least one processor 502 executes the sound effect vibration control method provided in this application embodiment.
[0097] Furthermore, the electronic device also includes a vibration feedback device 503, which is used to vibrate synchronously with the sound effect vibration signal output by the processor 502 to achieve the corresponding vibration effect.
[0098] Please refer to Figure 6, which is a block diagram of a computer-readable storage medium provided in an embodiment of this application.
[0099] As shown in Figure 6, this application embodiment also provides a computer-readable storage medium 600, which stores executable instructions 610. When the executable instructions 610 are executed, they perform the sound-effect vibration control method provided in this application embodiment.
[0100] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0101] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).
[0102] It should be noted that the various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For product-related embodiments, since they are similar to method-related embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method-related embodiments.
[0103] It should also be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0104] The above description of the disclosed embodiments enables those skilled in the art to implement or use the content of this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for controlling vibration in response to sound effects, characterized in that: Extract the psychological loudness features of the target audio data; wherein, the psychological loudness features include: the loudness value, instantaneous power peak, center frequency, and time of a single frame of audio data; Intensity information is calculated using the loudness value and the instantaneous power peak, and the center frequency is mapped to the center Mel frequency; A relative frequency curve is obtained based on the center Mel frequency of all the single-frame audio data, and a relative intensity curve is obtained based on the intensity information of all the single-frame audio data; A vibration signal that follows the sound effect is generated based on the relative frequency curve and the relative intensity curve. The vibration signal that follows the sound effect is sent to the actuator to drive the actuator to output corresponding vibration feedback.
2. The method for controlling vibration according to sound effects as described in claim 1, characterized in that, Prior to the step of extracting the psychological loudness features of the target audio data, the following steps are included: Read in the raw audio data with sound effects; The original audio data is preprocessed to obtain target audio data; wherein the preprocessing includes at least one of the following: filtering and downsampling.
3. The method for controlling vibration according to sound effects as described in claim 1, characterized in that, The step of obtaining the relative frequency curve based on the center Mel frequency of all the single-frame audio data includes: Obtain the relative frequency range of the actuator used to generate vibration feedback; Based on a preset first nonlinear function, the center Mel frequency of all the single-frame audio data is mapped to the relative frequency range to obtain a relative frequency curve; The step of obtaining a relative intensity curve based on the intensity information of all the single-frame audio data includes: Obtain the relative intensity range of the actuator used to generate vibration feedback; Based on a preset second nonlinear function, the intensity information of all the single-frame audio data is mapped to the relative intensity range to obtain a relative intensity curve.
4. The method for controlling vibration according to sound effects as described in claim 3, characterized in that, The method further includes: Based on a preset vibration sensing function, the relative intensity curve and the relative frequency curve are mapped to a vibration sensing curve; Based on the lower limit of vibration intensity that can be perceived by the human body, points in the vibration sensing curve that are lower than the lower limit are recorded as the first deletion points; Obtain the first corresponding time for all the first deletion points, and delete the points in the relative intensity curve and the relative frequency curve that are at the first corresponding time to obtain the optimized relative intensity curve and relative frequency curve.
5. The method for controlling vibration according to sound effects as described in claim 4, characterized in that, The step of mapping the relative intensity curve and the relative frequency curve to a vibration sensing curve based on a preset vibration sensing function includes: Obtain the relative intensity information contained in the relative intensity curve; The relative intensity information is weighted to obtain a weighted relative intensity curve; The relative frequency curve and the weighted relative intensity curve are mapped to a vibration sensing curve based on a preset vibration sensing function.
6. The method for controlling vibration according to sound effects as described in claim 4, characterized in that, The method further includes: The adjacent values of the target points in the vibration curve are checked. If the target points in the vibration curve meet the preset conditions, the target points are recorded as the second deletion points. Obtain the second corresponding time for all the second deletion points, and delete the points in the optimized relative intensity curve and the optimized relative frequency curve that are at the second corresponding time to obtain the simplified relative intensity curve and the simplified relative frequency curve.
7. The method for controlling vibration according to sound effects as described in claim 1, characterized in that, The step of generating a vibration signal that follows the sound effect based on the relative frequency curve and the relative intensity curve includes: An initial vibration signal following the sound effect is generated based on the relative frequency curve and the relative intensity curve. According to preset short signal parameters, a short signal is added to the initial sound effect vibration signal to obtain the final sound effect vibration signal; wherein, the short signal parameters include the short signal addition time, short signal frequency, and short signal strength.
8. The method for controlling vibration according to sound effects as described in claim 7, characterized in that, The method further includes: The peak points of the relative intensity curve are selected to obtain the first peak point; All the instantaneous power peaks are fitted into an instantaneous power peak curve, and the target point corresponding to the first peak point in the instantaneous power peak curve is obtained based on preset filtering conditions; The peak position of the first peak point is verified with that of the target point to obtain the short signal parameters.
9. The method for controlling vibration according to sound effects as described in claim 8, characterized in that, The step of filtering peak points of the relative intensity curve to obtain the first peak point includes: Peak points are selected from the relative intensity curve to obtain initial peak points; Connect all the initial peak points and smooth the curve to obtain the relative intensity envelope curve. Peak points are selected from the relative intensity envelope curve to obtain the first peak point; 10. The method for controlling vibration according to sound effects as described in claim 8, characterized in that, The step of fitting all the instantaneous power peaks into an instantaneous power peak curve and obtaining the target point corresponding to the first peak point in the instantaneous power peak curve based on preset filtering conditions includes: Obtain the corresponding time of all the first peak points, and combine it with the preset minimum time difference to obtain all the corresponding time intervals; Fit all the instantaneous power peaks to an instantaneous power peak curve, and obtain all points in the instantaneous power peak curve that fall within the corresponding time interval; All points within the corresponding time interval are recorded as target points corresponding to the first peak point.
11. The method for controlling vibration according to sound effects according to claim 10, characterized in that, The step of verifying the peak position of the first peak point and the target point to obtain the short signal parameters includes: The peak points of the instantaneous power peak curve are filtered to obtain the second peak point; Verify whether the second peak point exists among the target points; If the second peak point exists among the target points, then the first peak point and the target point are determined to have passed the peak position verification, and the first peak point that passed the verification is recorded as the target first peak point; Record the time corresponding to the first peak point of the target as the time when the short signal is added. The relative frequency corresponding to the time when the short signal is added is obtained on the relative frequency curve to obtain the short signal frequency. Obtain the relative intensity on the relative intensity curve at the time the short signal is added, and increase the relative intensity according to a preset relative intensity increase value to obtain the short signal intensity; Based on all the short signal addition times, the short signal frequency, and the short signal strength, the short signal parameters are obtained.
12. A vibration control device that responds to sound effects, characterized in that, include: The feature extraction module is used to extract the psychological loudness features of the target audio data; wherein, the psychological loudness features include: the loudness value, instantaneous power peak, center frequency, and frame time of a single frame of audio data; The feature mapping module is used to calculate intensity information using the loudness value and the instantaneous power peak, and to map the center frequency to the center Mel frequency; The parameter determination module is used to obtain a relative frequency curve based on the center Mel frequency of all the single-frame audio data, and to obtain a relative intensity curve based on the intensity information of all the single-frame audio data; The signal generation module is used to generate a vibration signal that follows the sound effect based on the relative frequency curve and the relative intensity curve; The driving module is used to send the vibration signal that follows the sound effect to the actuator to drive the actuator to output corresponding vibration feedback.
13. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the sound-effect vibration control method according to any one of claims 1 to 11.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the sound-effect vibration control method according to any one of claims 1 to 11.